Auto Archiver of Web URLs

A tool for automatically archiving web page links and saving the images and videos. Currently used in monitoring human rights situations in Ukraine, Myanmar and Afghanistan

Thanks to The Centre for Information Resilience for providing the datasets and to Bellingcat's excellent open source code to which we are contributing back.

As of 31st July 2022 we have archived 14,361 links totaling over 272GB of images and video which would have had to be done by hand. This is a huge time saver and we are proud to support the work of organisations like The Centre for Information Resilience

Given a list of Web URL links entered into a Google Spreadsheet, we archive that page, so that you have a safe copy.

Our archiver is automatic and every minute looks to see if there are links which need to be processed. Below shows the archiver working on some Twitter URL's which contain images.

The results of the archiver for row 4 in the spreadsheet above are:

• Gets Images from the Link
• Save the image to the Archive Location either Google Drive or Digital Ocean Spaces
• Save a Screenshot
• Save a Hash of everything for veracity
• Create a Thumbnail of the image
• Create a Upload timestamp

1. Twitter Image

https://twitter.com/minmyatnaing13/status/1499415562937503751 is the entered Link

Screenshot we took of that Tweet and saved to our cloud storage in case it is every deleted from Twitter.

Image embedded in the Tweet which we have saved to our cloud storage

Raw twitter data, link to saved image (with hash). And this page is hashed and result populated in the spreadsheet

2. Facebook Video

https://www.facebook.com/Karenni-Free-Burma-Rangers-111193108038684/videos/343188674422293/ is the entered Link

Showing the successful archiving of the link

Automated screenshot of the Facebook page.

video.mkv is the raw video downloaded from Facebook.

Thumbnails of parts of the video so can see at a glance what happened.

Archivers

We have specific archivers for

• Twitter
• YouTube
• Facebook
• Telegram
• TikTok

There is a fallback archiver (which uses archive.org) for any other site.

Conclusion

This tool can save a huge amount of manual time archiving data from sites. It is being used by organisations capturing media for safeguarding.

Please contact us here and we can do a demo for you, or we can add you to our demo spreadsheet, so you can test out the system in real time.