1. Etherna
  2. Blog
  3. Development
  4. Introduction: Etherna Video Importer

Federico Cicciarella

October 28, 2023
Development

Introduction: Etherna Video Importer

The Etherna Video Importer is a command-line application designed to empower users across Linux, Windows, and macOS platforms to effortlessly import videos from YouTube or other custom sources. With its seamless integration, users can then upload these videos to the Etherna platform.

Download

Download the Etherna Video Importer from releases page on GitHub.

FFmpeg and ffprobe

The application requires FFmpeg and ffprobe. Often, the two are distributed together. If you have an FFmpeg installation present on your system, it will be used. Otherwise, you can download them from https://ffmpeg.org/ and copy them inside the <app_dir>/FFmpeg folder for use it during the import, or you can pass their location as arguments.

General options:

-k, --api-key: The API Key is the authentication key used for uploading and listing videos in the index. If provided, it will bypass browser authentication.

-f, --ffmpeg-path: Specifies the path to the FFmpeg and ffprobe folder. If not specified, the application will search for them in <app_dir>/FFmpeg, and if not found there, it will look in the global installation. It's crucial that FFmpeg and ffprobe are present in one of these paths; otherwise, the importer will produce an error.

-i, --ignore-update: When the importer is executed, it checks for updates from GitHub. If found, it will recommend an update and block the process. Having the latest version is always recommended. However, if interruptions need to be avoided, this option can be used to ignore this update check.

-a, --auto-purchase: Automatically accepts the purchase of all batches. Each uploaded video is associated with a batch and incurs a cost based on its duration. The importer will request purchase confirmation if this option is not selected. The batch cost will be charged to the authenticated user.

-t, --ttl: Specifies the Time-to-Live (TTL) of postage batches (default: 365 days). This indicates the duration of the batch to be purchased, with longer durations incurring greater costs in terms of BZZ.

-o, --offer: Offer video downloads to everyone. In this case, the cost of viewing the video will be charged to the owner of the video.

-p, --pin: Pins videos on Etherna’s servers. This is currently strongly recommended.

--force: Forces the upload of videos even if they have already ben uploaded. This option can be useful in case a previously uploaded video needs to be overwritten due to defects.

-m, --remove-missing: Removes indexed videos generated with this tool but missing from the origin source.

--remove-unrecognized: Removes indexed videos not generated with this tool.

-u, --unpin: Attempts to unpin content that has been removed from the index.

Import Youtube Single Video

In our example, we will download the video 'Ethereum in 30 minutes by Vitalik Buterin | Devcon Bogotá'. To do this, we will connect to YouTube to obtain the video's URL.

And here is our URL copied directly from the browser: ​link

To start the import, execute the following command:

./evi youtube-video https://www.youtube.com/watch?v=UihMqcj-cqc

If you haven't provided an API Key, the browser will open for you to log in to Etherna. The user you log in with will be used for the import.

Enter your login credentials. At this point, the import process will begin, and you can monitor the progress by returning to the console application.

You can check the progress directly from the console

Import Youtube Channel

To import all the videos of a youtube channel retrieve the url of the channel

In our example the channel to download is ​https://www.youtube.com/@EthereumFoundation so we will launch the importer like this

./evi youtube-channel https://www.youtube.com/@EthereumFoundation

All the videos within the channel will be imported, and you will be able to see the download list directly from the console

Bulk Import from Json

Through JSON data upload, it is possible to import video sources from either local files on your computer or remote URLs sources. Additionally, you can specify an image to be used as a thumbnail, which can be uploaded locally or fetched from a URL. It's important to note that the Thumbnail field is optional and can be left empty. The other mandatory fields include Id, Title, and Description. The Id field will be used as a key to check if the data is already present in our index. If it already exists, the old data will be modified; otherwise, a new one will be added to the index as new.We will now provide sample JSONs showing how to correctly fill in the required fields, and how it is possible to use both local and remote resources. 

  json
  
 [ 
     { 
         "Id": "ID_00001", 
         "Title": "Ethereum in 30 minutes by Vitalik Buterin", 
         "Description": "Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.", 
         "VideoFilePath": "/home/fedec87p/Etherna/Demo_Importer_Json/Ethereum in 30 minutes by Vitalik Buterin.mp4", 
         "ThumbnailFilePath": "https://fastly.picsum.photos/id/1/5000/3333.jpg?hmac=Asv2DU3rA_5D1xSe22xZK47WEAN0wjWeFOhzd13ujW4" 
     }, 
     { 
         "Id": "ID_00002", 
         "Title": "Composability and Gamification", 
         "Description": "Web3 tech provides features that have been practically unexplored by game designers. Games use DeFi mechanics, or self-custody and P2P marketplace mechanics, but few have used composability to create new game experiences for incentivization.", 
         "VideoFilePath": "Composability and Gamification.mp4", 
         "ThumbnailFilePath": "" 
     }, 
     { 
         "Id": "ID_00003", 
         "Title": "Sample Stream m3u8", 
         "Description": "This is a sample for download a stream m3u8 with URL", 
         "VideoFilePath": "https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/.m3u8", 
         "ThumbnailFilePath": "thumb.jpg" 
     } 
 ]

To initiate the import, you will need to specify the path where the configuration JSON is located. In my example, the file is situated in the directory /home/fedec87p/Etherna/Demo_Importer_Json/ and is named VideoList.json. Therefore, I will execute the following command:

./evi json /home/fedec87p/Etherna/Demo_Importer_Json/VideoList.json

With this example we will import three videos.

  1. For the first video, the VideoFilePath is specified as an absolute URL located on your computer, while the image for ThumbnailFilePath is obtained from a remote URL..
  2. The second video uses a relative path, which means that the "Composability and Gamification.mp4" file should be in the same folder where the import JSON was selected (/home/fedec87p/Etherna/Demo_Importer_Json/Demo_Importer_Json). The ThumbnailFilePath field has been left empty, so no image will be loaded.
  3. In the third video, a remote m3u8 URL is specified, while the image is specified as relative, so the thumb.jpg file should be located at (/home/fedec87p/Etherna/Demo_Importer_Json/Demo_Importer_Json).

The console will display the progress of the import.

Bulk Update from Json

Suppose we have previously uploaded videos with IDs ID_00001, ID_00002 and ID_00003 but we wish to modify the title or description instead of deleting the video from the index and re-uploading it. In this case, it is possible to update the index data. 

  json
  
 [ 
     { 
         "Id": "ID_00001", 
         "Title": "Ethereum in 30 minutes (EDIT)", 
         "Description": "Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.", 
         "VideoFilePath": "/home/fedec87p/Etherna/Demo_Importer_Json/Ethereum in 30 minutes by Vitalik Buterin.mp4", 
         "ThumbnailFilePath": "https://fastly.picsum.photos/id/1/5000/3333.jpg?hmac=Asv2DU3rA_5D1xSe22xZK47WEAN0wjWeFOhzd13ujW4" 
     }, 
     { 
         "Id": "ID_00003", 
         "Title": "Sample Stream m3u8", 
         "Description": "This is a sample for download (EDIT)", 
         "VideoFilePath": "https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/.m3u8", 
         "ThumbnailFilePath": "thumb.jpg" 
     } 
 ]

In this manner, the video with ID_00001 in the index will undergo a modification of the manifest, changing the title from Ethereum in 30 minutes by Vitalik Buterin to Ethereum in 30 minutes (EDIT) . Similarly, the video with ID_00003 will experience a modification of the description from This is a sample for download a stream m3u8 with URL to This is a sample for download (EDIT).

ID_00002 will remain unchanged since it is not included in the list, unless the --remove-missing command is specified. In such a case, if it's not present in the data source, it will be removed. Regardless, any unmodified video will be skipped.

Please note that when dealing with an existing ID, only the Title and Description fields will be modified. If you want to upload VideoFilePath or ThumbnailFilePath again, you need to specify the --force command.