Friday, February 28, 2025

New UFO timeline tool: 80,000 page chronological document and collection of 26,000 JSON files summarising posts on main UFO subreddits

As part of updating and expanding the 1,700 page UFO timeline I circulated in 2006 (created by cross-referencing material in about 900 UFO books that I read in 2004-2006), I recently shared online a 5,000 page timeline of UFO news articles from 2005 onwards.

Of course, media stories about UFOs only contain a fraction of the available information about UFOs and its weird and wonderful sub-culture (or rather sub-cultures, plural).

As the latest in a series of tools to help locate and assimilate relevant information about UFOs from other sources (ranging from UFO podcasts and UFO documentaries to PhD dissertations about UFO, academic journal articles about UFO and university archives among other material), I recently turned to Reddit.

I originally had in mind a short mini-project in relation solely in relation to posts on the largest UFO sub-reddit, r/UFOs (which currently has approximately 3.3 million members) at: https://www.reddit.com/r/UFOs/

However, thanks to extremely useful technical tips from French researcher Laurent Chabin of the French UFO group SCEAU, I was able to speed up part of the relevant process significantly (about 100 times faster...).

So, of course, my immediate reaction was to expand the scope of the work I was doing. :)

After a bit more time and effort than I originally intended (and more than a little bit of mission-creep), I have used the techniques helpfully suggested by Laurent Chabin of SCEAU to generate:

(1) PDF documents setting out the posts on various UFO subreddits (listed below). These summary documents range from a few pages to a few tens of thousands of pages.

(2) A combined chronological document setting out the relevant posts. This one is about 80,000 pages long.

(3) A set of JSON files for anyone else interested in further work on this set of data (and potentially for use in my planned future UFO projects, since it should be fairly trivial for to combine this set of data with the data from the media sources - and other material as well...).


COMBINED DOCUMENT: 80,043 page chronology summary of UFO posts on Reddit at:

This includes a summary of many of the posts on most of the main UFO subreddits on Reddit, including those listed below.  Due to some technical issues and prohibitions by Reddit, it does not include all the posts on those groups (particularly the older ones) - but (as indicated by the page sizes involved) it does include quite a few!

Some of the data obtained on the Reddit posts is also incomplete or out of date, particularly the "score" (number of upvotes minus the number of downvotes) on each post and the number of replies to each post.  The problem largely seems to be that the numbers are based on when the Reddit post was archived, so I have added a prefix of "Old" to those columns e.g. "Old score".  (I did consider omitting those columns altogether, but during my use of these summary documents I found it useful to have an indication - even if incomplete or out of date - of at least some of the posts that had a relatively high score or high number of replies.  If anyone prefers to avoid out of date information, they can delete those columns - but simply keeping this point in mind should be a sufficient precaution). 



AcademicUAP subreddit at:
Approximate membership: 3,700 


Aliens subreddit at:
https://www.reddit.com/r/Aliens
Approximate membership: 1 million 


DisclosureParty subreddit at:
https://www.reddit.com/r/DisclosureParty
Approximate membership: 23,000


Experiencers subreddit at:
Approximate membership: 87,000


Reddit_UAP_Database subreddit at:


SETI subreddit at:
https://www.reddit.com/r/SETI
Approximate membership: 17,000


The_UFO_Phenomenon subreddit at:


UAP subreddit at:
https://www.reddit.com/r/UAP
Approximate membership: 462


UFO subreddit at:
https://www.reddit.com/r/UFO
Approximate membership: 353,000


UFOB subreddit at:
https://www.reddit.com/r/UFOB
Approximate membership: 190,000


UFObelievers subreddit at:
https://www.reddit.com/r/UFObelievers
Approximate membership: 127,000


UFOBookClub subreddit at:
https://www.reddit.com/r/UFOBookClub
Approximate membership: 3,300


UFOdocumentaries subreddit at:
https://www.reddit.com/r/UFOdocumentaries
Approximate membership: 87,000


UFOPilotReports subreddit at:
https://www.reddit.com/r/UFOPilotReports
Approximate membership: 18,000


UFOs subreddit at:
https://www.reddit.com/r/UFOs
Approximate membership: 3.3 million


UFOscience subreddit at:
https://www.reddit.com/r/UFOscience
Approximate membership: 135,000


UFOstudies subreddit at:
https://www.reddit.com/r/UFOstudies
Approximate membership: 911


UFOtruth subreddit at:
https://www.reddit.com/r/UFOtruth
Approximate membership: 17,000


USOs subreddit at:
https://www.reddit.com/r/USOs
Approximate membership: 3,700


There are, of course, posts about UFOs in numerous other subreddits on Reddit not focused on UFOs - ranging from wider paranormal groups, to groups touching on topics overlapping with areas of interest to UFO researchers and local groups which sometimes have a post about a UFO sighting in their own town/city/country.  I'll be posting some additional summary documents soon in relation to some of those other groups - but I think the list below includes the main groups focused on UFOs/UAP.  (I did ask on various social media platforms if I had missed any and added a few groups in the light of the helpful input I received).




The above graphic illustrates just some initial steps (and simplifies those steps a bit, for ease of illustration...).  Most of those steps are now complete, or at least reached the point of diminishing returns.   

I expect the next stages, which build on this new foundation, to focus on deeper dives on particular issues/cases and resolving some of the disputes that have arisen in relation to them i.e. a bit of fun.... 

Thursday, February 27, 2025

Huge data dump on USOs (Unidentified Submerged Objects), UFOs and water - Part 1 - Marco Bianchini "USOCAT" catalogue, Richard Dolan spreadsheet, Reddit "USOs" post collection, Jan Aldrich's NavCat, "Russia's USOs Secrets" etc

Introduction

As outlined (with links) in the sections below, I have a few treats for anyone interested in reports of "USOs" (Unidentified Submerged Objects)...

Italian USO / UFO researcher, Marco Bianchini, has kindly given me permission to upload a copy of his "USOCAT" catalogue of Italian USO cases. Links below.

This week, Richard Dolan also helpfully shared on his website a PDF of his spreadsheet of USO sightings. I include a link below to that document on his website and also include code I've generated so that anyone interested can easily "reverse engineer" that PDF back into an Excel spreadsheet. (I'm not uploading a copy of the Excel file itself as I haven't received permission to add it).

I've also created a chronological summary (125 pages) of Reddit posts to the "USOs" subreddit.

As an experiment with the recently released "Deep Research" tool from ChatGPT, I have uploaded a report generated automatically by that tool on the USO topic.  

Finally, I've uploaded a "podcast" generated by AI software after being fed the book "Russia's USO Secrets" by Paul Stonehill and Philip Mantle (with permission from both of those authors).











(1) Marco Bianchini "USOCAT" catalogue 

USO / UFO researcher, Marco Bianchini, is a member of Italian UFO group CISU ("Centro Italiano Studi Ufologici"): 
https://www.cisu.org/

Marco has kindly given me permission to upload a copy of his "USOCAT" catalogue to my online archive hosted by the AFU in Sweden.  USOCAT provides details of reported observations of USOs in Italy's rivers, lakes and territorial waters:
https://files.afu.se/Downloads/?dir=.%2FDatabases%2FMarco%20Bianchini%20USOcat


(I really should find the time to attempt to get permission to upload various other UFO/USO catalogues and databases.  I now have over 70 in my off-line collection and some of their creators may be willing for me to upload them, although I know some will not...). 

Marco has also helpfully given permission me to add a copy of his list of books on the USO topic in chronological order. I think this document is useful for anyone doing any deep dives (no pun intended...) into USOs. I have included that further document from Marco in the same folder as his USOCAT, at the link above



(2) Reddit "USOs" post collection

I've also created a chronological summary (125 pages) of Reddit posts to the "USOs" subreddit.

That PDF can be downloaded here:
https://files.afu.se/Downloads/Websites/Reddit/Subreddit%20-%20USOs/Koi%2C%20Isaac%20-%20Chronology%20of%20USOs%20Subreddit%20Posts.pdf






(3) Automated podcast about the book "Russia's USO Secrets"

I recently wrote about some of my recent experiments to apply Artificial Intelligence to UFO research by using an AI tool to generate podcast-style discussions of various UFO books, scientific papers and UFO documents. The voices of the podcast “hosts” are AI generated, as is all the discussion. I thought the results were interesting and potentially useful (although obviously there is a lot of room for improvement, which I anticipate will occur fairly rapidly…). 

I’ve now uploaded to Youtube link below a further such “podcast” which discusses a book on Russian USOs, "Russia's USO Secrets" by Paul Stonehill and Philip Mantle.

https://youtu.be/lLo8HtIuI5w




One of the co-authors of that book, Paul Stonehill, also has various videos on Youtube about Russian USO reports, including the following:











(4) Jan Aldrich's NavCAT

Jan Aldrich of Project 1947 has previously shared on his website a "draft" catalogue of "UFOs/USOs Report by Seagoing Services".  

Jan has previously given me permission to archive the entirety of the Project 1947 website, so I've now added a PDF copy of the NavCat catalogue to my online folder of UFO/USO catalogues and databases, specifically at:




(5) ChatGPT "Deep Research" on USOs 

As an experiment with the recently released "Deep Research" tool from ChatGPT, I have uploaded a report generated automatically by that tool on the USO topic:
https://files.afu.se/Downloads/Websites/AI/2025%20-%20ChatGPT/o3%20Deep%20Research/0%20-%20Other/USOs/AI%20-%202025%20-%20o3%20Deep%20Research%20-%20USOs%20-%20Unidentified%20Submerged%20Objects.pdf

I think the results are sufficiently interesting to be worth expanding upon in a separate mini-project, which I hope to post within the next day or two.





(6) Richard Dolan's USOs spreadsheet

Earlier this week, Richard Dolan helpfully shared on his website a PDF of his spreadsheet of USO sightings.  That spreadsheet includes it includes the data for all three volumes of his books "A History of USOs".

Here is some code I've generated so that anyone interested can easily "reverse engineer" (yes, UFO pun intended...) that PDF back into an Excel spreadsheet (or CSV file etc) so that its content can be assimilated (yes, Star Trek pun intended...) more easily into a unified collection of various UFO catalogues and databases. 

(I'm not uploading a copy of the Excel file itself as I haven't received permission from Richard Dolan to add it to my online archive).


import pdfplumber
import csv

def extract_complete_sheet(pdf_path):
    """
    Extracts the COMPLETE sheet table from the PDF using pdfplumber's extract_table() method.
    It looks for pages containing the marker "USO History Spreadsheet Data COMPLETE" and
    collects table data from those pages. Duplicate header rows (starting with "DATE") are skipped.
    Any semicolons in cells are removed.
    """
    complete_data = []
    header = None

    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            page_text = page.extract_text() or ""
            # Process only pages that mention the COMPLETE sheet
            if "USO History Spreadsheet Data COMPLETE" in page_text:
                table = page.extract_table()
                if table:
                    for row in table:
                        # Skip rows that are completely empty
                        if not any(cell and cell.strip() for cell in row):
                            continue
                        # Remove semicolons from each cell (if cell is None, use empty string)
                        clean_row = [cell.replace(";", "") if cell else "" for cell in row]
                        # Identify header rows by checking if the first cell starts with "DATE"
                        if header is None and clean_row[0].strip().upper().startswith("DATE"):
                            header = clean_row
                            complete_data.append(header)
                        else:
                            # Skip duplicate header rows
                            if header and clean_row[0].strip().upper().startswith("DATE"):
                                continue
                            complete_data.append(clean_row)
    return complete_data

def write_csv(data, output_csv):
    """
    Writes the extracted table data to a CSV file using semicolons as delimiters.
    """
    with open(output_csv, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f, delimiter=";")
        for row in data:
            writer.writerow(row)

if __name__ == "__main__":
    pdf_path = "richard-dolan-uso-history-spreadsheet-data.pdf"  # Relative path (file in same folder)
    output_csv = "dolan USO spreadsheet.csv"                      # Output file in same folder

    data = extract_complete_sheet(pdf_path)
    if not data:
        print("No data extracted. Please verify that the PDF contains the expected COMPLETE sheet table.")
    else:
        write_csv(data, output_csv)
        print(f"CSV file has been saved as '{output_csv}'")

This code was created using a free online AI tool (Grok 3) in a matter of minutes.  My main interest in sharing this code is to stimulate thinking about use of AI to generate many sets of similar code to assimilate data from individual UFO books and UFO articles into structured electronic formats and (permissions allowing...) potentially assimilating that data into more comprehensive collections.








Wednesday, February 12, 2025

Collection of VERY rare 1950 UFO newsletters (USA) completed, thanks to someone going to a local library

4 years ago, I made scans of most issues of the _RARE_ Grand Rapids Flying Saucer Club's newsletter "UFORUM" (1950s, USA) freely available online - with thanks to Barry Greenwood for the scans.

At that time, Issues 1 to 3 were missing from the huge UFO archives of both the AFU and Barry Greenwood (and every other UFO archives with whom I cooperate...).

Today, Bradley W Plaisier (@bpleasies on Twitter) filled in those gaps after he stopped by the Grand Rapids public library while he was in town and using his mobile phone to take images of those issues.

The collection of "UFORUM" is therefore now complete and in my online folder for UFO magazines / newsletters from the USA.

Many gaps in UFO research could be filled relatively quickly and easily if a few more people were willing to go to specific archives / libraries (particularly in the USA) to take a some photos of relevant correspondence, UFO documents and rare UFO newsletters...

AI has helped identify gaps. Now just need to fill them.




Tuesday, February 11, 2025

New 5,000+ page timeline of UFO news articles (2005 onwards) ... plus 7,000 related JSON files

I have now uploaded a new UFO timeline tool: a 5,000+ page chronological summary of online UFO news articles from 2005 onwards, with links to those articles.  (I have also uploaded some new related collections of thousands of files, as outlined briefly below).

This document is part of my recent effort to use AI tools to impose more order on my various collections of UFO material (and to help fill in some of the gaps in those collections).  I am also using this document as one of the tools for updating and revising my previous UFO Timeline that I shared online in 2006

I'm happy to make this new UFO timeline document freely available for anyone to download.

This new document seeks to collate information from Google News and to generate structured data based on it (in the form of a collection of about 7,000 JSON files, which I have now also shared online).  The JSON files are then used to generate a chronological table. 

I have also been experimenting with using the links in all of the JSON files to download all of these news articles and name them using a consistent filename format, but this last part in rather in a work in progress. I've shared a sample of about 1,000 PDFs to give a brief indication of the current state of this effort. (The current problems relate mainly to the scraping code I've generated using AI to properly handling cookie consent popups and related - hopefully minor - necessary improvements to my current  code before it is deployed on a much larger scale) 

(I've included links above to the 5,000 summary document, the 7,000 JSON files and the sample of 1,000 PDFs).

I'm planning on extending this project about online news articles during the next few days and then combining these relatively recent news articles with my existing collection of about 100,000 older scanned newspaper clippings about UFOs from the 1940s onwards (collated during the last couple of decades from working with over 100 different UFO groups/researchers and combining their collections). 


 




My previous UFO timeline/chronology (1,700 pages based on reading about 900 UFO books)

Back in 2006, I circulated among the UFO community a UFO timeline/chronology that I wrote by cross-referencing material in about 900 UFO books that I'd read in 2004-2006.

It was about 1,700 pages long (much of which was taken up with references).

That chronology gave the dates of various frequently discussed events, incidents, individuals and documents in the history of ufology and is cross-referenced to related material.

I  concentrated on references to discussions in UFO and SETI books. As part of the preparation of that document, I read a little over 1,000 such books noting relevant discussions.

Here's a link to a free PDF version of that timeline (and I have also made available a Microsoft Word copy for anyone wishing to edit/revise it):
https://files.afu.se/Downloads/Books/Other/Koi%2C%20Isaac%20-%20UFO%20Chronology%20%281877-2006%29.pdf


Back then, I was a bit naive and hoped that the UFO community could work together to develop the free resource that I shared. :)

Almost 20 years have passed since then.

I've recently been doing some work on a major update and revisions to the original timeline - this time assisted by AI/RAG tools which are making the work more effective and efficient...

I originally had in mind preparing a chronology about 20-30 pages long. The Chronology quickly became, um, considerably longer.

This chronology was intended to act as an efficient way of organising references and hyperlinks to relevant material, in the interests of more fully informed debate and reducing the amount of reinvention of the wheel.

Under the present civil litigation procedure in England in large commercial cases, the parties to litigation are expected to co- operate in various respects so as to make a case fit for trial in as efficient and fair a manner as possible. This commonly involves, at the most basic level, producing various agreed (and therefore non-contentiously drafted) documents so that the trial judge will be able to understand the nature of the dispute and the issues in dispute as efficiently as possible. These non- contentious documents typically include:

(1) a chronology (giving the dates of key events, a few words to identify the event and cross references to further material relied upon by both sides in relation to that event;

(2) a List of Issues (listing the main issues agreed and not agreed between the parties);

(3) a case memorandum (summarising the case in just a few pages);

(4) a dramatis personae (listing the names that the Judge will come across at trial, with a few words to identify them);

(5) a glossary (listing technical terms or terms of art that will arise during the trial, with an agreed definition of those terms);

(6) a reading list (of material to be read in advance of the trial).

I believe that ufology would greatly benefit from adopting some elements of this approach. All too often, the debates regarding UFO reports do not focus on the real issues (or even identify the issues which should be addressed).





Wednesday, October 30, 2024

UFO archive links - 13,552 page overview searchable PDF

I have been doing some work to make it easier to find items in my online UFO archive kindly hosted by the AFU in Sweden, including creating some new AI tools and also some new documents.

I've added a few million pages of material to that archive in the last couple of years (including more scans of UFO magazines/newsletters, official documents, dissertations, transcripts etc). This means that the archive is becoming an increasingly useful resource, but also makes it harder to quickly locate all relevant items.

As one basic - but I think useful - new tool, I've used Python code (generated with the assistance of an AI chatbot) to compile a basic list of the files and folders in my online archive - with hyperlinks to each item.

Because the content of this list is limited just to folder/file names (and a link to that folder/file) without any additional details, the new PDF is relatively short searchable overview (just 13,552 pages).   

[Even this limited document can take a few seconds to download, as it is a 131MB file].

Once this single PDF is downloaded, you can quickly search the titles of the files (but not - currently - their content) for a keyword in those titles and the results will include a hyperlink to the relevant files. (I've written about performing offline searches of the entire archive and may add a full online search function at some point).

I'm currently torn between adding more to the archive or focusing on writing up the results of some deep dives I've been doing on specific cases/topics.  I'm increasingly focusing on the deep dives.



The single PDF is at the link below:

https://files.afu.se/Downloads/0%20-%20index%20koi%20-%20overview%20list%20of%20archive.pdf


That PDF can also be found at the bottom of the main page of the archive, i.e. at the bottom of: 

https://files.afu.se/Downloads/ 




Sample snapshots of parts of the results for quick searches of this single PDF (using the fast and free PDF-Xchange Editor, but similar searches can be done using virtually any PDF software):

"Australia":




"Wales":

  


Monday, September 2, 2024

PDFs: "UFO Encounters" newsletter (Michael Norris, USA, 1990s)

Michael Norris, formerly of MUFON Georgia, edited the "UFO Encounters" newsletter in the 1990s. He became webmaster for the "International Society for UFO Research" ("ISUR") later in the 1990s.

I tried to obtain contact details for Michael Norris by posting on Facebook and elsewhere in 2016, again in 2017 and again in 2021. Those attempts failed.

Fortunately, I was contacted recently by someone that knows Michael Norris. He not only had an email address for Michael but had also taken the time to speak to Michael about my archiving project, and Michael was amendable to an upload.

Michael Norris has kindly confirmed to me that I have his permission to make searchable scans of his "UFO Encounters" magazine freely available online. 

15 issues (scanned by the AFU in Sweden) are now available online by clicking on the image below:



I have added these issues to my folder for UFO magazines/newsletters from the USA.