code

Background

Let’s start by saying this was just a fun project that I wanted to create in order to have something new as part of my portfolio. Since every single website that I know about that provide this type of service are still made with PHP and SQL, I decided to use JavaScript which nowadays is the to-go development tool that everyone seems to be using.

The functionalities are simple, find and fetch data from a given YouTube URL and displays it to the front-end user. What I needed for this project was three sections, a form with a text input; a left div containing the video that was being fetched and a right div that is going to work as the container for all the videos previously retrieved – so the user can go back when he/she is ready to download X video but… – by the user.

All of that is mainly for the front-end side of the application which I’ll talk about later on. Right now, let’s talk about the server which makes all of this possible!.

Setting up the Server

Assuming you’re working on a pre-made project, please install the npm ytdl-core library by using the following command:

npm i ytdl-core

Furthermore, we now have the need to make use of it and for us to make it happen, we will create several HTTP endpoints.

const express = require('express');

const {
  getYouTubes,
  getYouTube,
  createYouTube,
  deleteAllYoutube
} = require('../../../controllers/extras/youtube');

const router = express.Router();

/*
* https://yourwebsite.com/api/v1/extras/youtube
*/
router.route('/').get(getYouTubes);

/*
* https://yourwebsite.com/api/v1/extras/youtube/getinfo
* This one is a POST request meaning that it can receive data from the front-end
*/
router.route('/getinfo').post(createYouTube);

/*
* https://yourwebsite.com/api/v1/extras/youtube/deleteall
* This is the function that runs the 15 of each month with a cron-job configured
*/
router.route('/deleteall').delete(deleteAllYoutube);

/*
* https://yourwebsite.com/api/v1/extras/youtube/0123/4567
* When accessing a single video page, the  data will be retrieved from this endpointt
*/
router.route('/:id/:videoid').get(getYouTube);

module.exports = router;

The code above contains some comments in which I tried to give a summary about what the purpose of each endpoint will be about. Obviously, this might be a bit confusing for beginners who are just getting into the NodeJS enviroment but believe me once you have created an endpoint, understanding the concept of REST APIs is waaaay toooooo easyyyyy.

How does our logic code looks like?

In computer science and IT in general when someone says the infrastructure and/or logic they are talking about the process that handles the requests or X function.

A good example is our first route; said route only needs to fetch the data from our database without any type of filtering and the perfect way to achieve that is by simply setting our YouTube model and calling its find() method; obviously if you want to apply some filtering then you will need to edit your find() into something like find({ x: y }) where X is the name and y is the value you want to find in X. Take a look into what it looks like:

// @desc    Get all youtube
// @route   GET /api/v1/extras/youtube
// @access  Private
// @status  DONE
exports.getYouTubes = asyncHandler(async (req, res, next) => {
  const youtube = await YouTube.find();
  res.status(200).json({ success: true, data: youtube });
});

Secondly, as a developer you already know what you’re looking for if you’re reading this post. I’m assuming what you need to know is the function that makes the YouTube app work, right? Well, I’ll put it easy; that and because I do not want to make this post longer.

As mentioned before the ytdl-core is the library being used to retrieve data from the YouTube data.

// @desc    Get youtube video data
// @route   GET /api/v1/extras/youtube/getinfo
// @access  Private
// @status  DONE
exports.createYouTube = asyncHandler(async (req, res, next) => {
  /*
   *
   * req.body usually contains the data coming from the front-end, commonly via a form/input element.
   * It is also important to note that said data comes according to the name of the input
   * e.x: <input name="password" type="password" value="THIS_IS_MY_PASSWORD" />
   * e.x2: <input name="title" type="text" value="THIS_IS_MY_TITLE" />
   *
   * The way to retrieve them on express with req.body would look likes this
   * const { password, title } = req.body
   * or
   * req.body.password and req.body.title. Each of these lines will then save
   * the data( THIS_IS_MY_PASSWORD and THIS_IS_MY_TITLE) into a temporary memory.
   */
  const videoUrl = req.body.video_url

  // If not videoUrl given, throw an error - self-explanatory
  if (!videoUrl) {
    return next(new ErrorResponse(`Please add a Url to shorten`, 400))
  }

  // Check videoUrl is valid - not necessary but this just assures the given URL is properly formatted.
  if (!validUrl.isUri(videoUrl)) {
    return next(new ErrorResponse(`Invalid URL`, 400))
  }

  // Validate YouTubeURL
  ytdl.validateURL(videoUrl)

  /*
   *
   * Once the URL validation has turned out to be "green", then we can proceed to the following function of getinfo() which as titled,
   * takes and gets the info regarding the video of given URL
   *
   */
  // Get info from Video
  const video = await ytdl.getInfo(videoUrl)

  /*
   *
   * Once the video has been found, I now want to retrieve all the formats that are played as videos by using the function
   * filterFormats and store the data into memory within a variable called videoFormats;
   *
   */
  // Find ONLY video NO AUDIO formats to choose from
  const videoFormats = ytdl.filterFormats(video.formats, 'videoonly')

  /*
   *
   * From all those video formats found, I now need the best quality one.
   * The chooseFormat() function which takes two parametes with the first one being the formats and
   * a second parameter which specifies the quality will make this possible.
   *
   */
  // Find the highest ONLY video NO AUDIO to download
  const highQualityOnlyVideo = ytdl.chooseFormat(videoFormats, {
    quality: 'highest',
  })

  /*
   *
   *
   * With that being said, if for some reason you want to generate links to download the audio
   * from a given video, we can again make use of the filterFormats() function and this time, look for audioonly
   *
   *
   */
  // Find ONY audio NO VIDEO formats to choose from
  const audioFormats = ytdl.filterFormats(video.formats, 'audioonly')

  /*
   *
   *
   * Found the audio? thats great, now find the best quality audio available
   *
   *
   */
  // Find the highest ONLY audio NO VIDEO to download
  const highQualityOnlyAudio = ytdl.chooseFormat(audioFormats, {
    quality: 'highest',
  })

  /*
   *
   *
   * Lastly the final video with the best audio and visuals found.
   *
   *
   */
  // Find the video to download
  const finalVideo = ytdl.chooseFormat(video.formats, {
    quality: 'highest',
  })

  res.status(200).json({
    success: true,
    data: video,
  })
})
;

This in fact might be great but it puts too many restrictions in our end; these include having to make several request to the same URL and/or having to download the files in one sitting even if you’re not ready yet. This is reason, I usually like to store data in my own db even if not really necessary. Making this is easy, we only need to set a create() method using our YouTube model:

// Create document
const youtube = await YouTube.create({
  videoId: video.videoDetails.videoId,
  title: video.videoDetails.title,
  text: video.videoDetails.description,
  videoFetchedUrl: video.videoDetails.video_url,
  videoEmbedUrl: video.videoDetails.embed?.iframeUrl,
  videoToDownload: {
    url: finalVideo.url,
  },
  videoOnly: {
    url: highQualityOnlyVideo.url,
  },
  audioOnly: {
    url: highQualityOnlyAudio.url,
  },
  thumbnails: video.videoDetails.thumbnails.map((image) => image.url),
  author: {
    name: video.videoDetails.author?.name,
    link: video.videoDetails.author?.user_url,
    channelUrl: video.videoDetails.author?.channel_url,
  },
  category: video.videoDetails.category,
  likes: video.videoDetails?.likes,
  dislikes: video.videoDetails?.dislikes,
  keywords: video.videoDetails.keywords
    ? video.videoDetails.keywords
        .map((keyword) =>
          keyword.trim().replace(/ +/g, ' ').split(' ').join('-').toLowerCase()
        )
        .filter((keyword) => keyword.length !== 0)
    : 'no-tags',
  views: video.videoDetails?.viewCount,
})

The Final Step

Let’s say that whenever we make a request to ytdl.getInfo(url) most of the info we need is found within an object called videoDetails which contains said info and way more data that might be good to your project. Now that’s the usual way to store data into a db and now instead of returning the video object, we will return the data that was saved in our db by typing youtube in the data variable that is located inside our json object:

res.status(200).json({
  success: true,
  data: youtube
})

Now that’s pretty much everything you need to know. If you already know about how mongoose works with MongoDB, you can simply copy and paste the functions and models below into your project; you can also modify the endpoints to fit your own needs.

The To-Copy-Paste Code

const ytdl = require('ytdl-core');
const validUrl = require('valid-url');
const YouTube = require('../../models/extras/YouTube');
const ErrorResponse = require('../../utils/ErrorResponse');
const asyncHandler = require('../../utils/async');

// @desc    Get all youtube
// @route   GET /api/v1/extras/youtube
// @access  Private
// @status  DONE
exports.getYouTubes = asyncHandler(async (req, res, next) => {
  const youtube = await YouTube.find();
  res.status(200).json({ success: true, data: youtube });
});

// @desc    Get single youtube
// @route   GET /api/v1/extras/youtube/:id/:videoId
// @access  Private
// @status  DONE
exports.getYouTube = asyncHandler(async (req, res, next) => {
  const youtube = await YouTube.findOne({
    _id: req.params.id,
    videoId: req.params.videoid
  });
  if (!req.params.id.match(/^[0-9a-fA-F]{24}$/) || !youtube) {
    return next(
      new ErrorResponse(`Document not found with id of ${req.params.id}`, 404)
    );
  }
  res.status(200).json({ success: true, data: youtube });
});

// @desc    Get youtube video data
// @route   GET /api/v1/extras/youtube/getinfo
// @access  Private
// @status  DONE
exports.createYouTube = asyncHandler(async (req, res, next) => {
  const videoUrl = req.body.video_url;

  // If not videoUrl given, throw an error
  if (!videoUrl) {
    return next(new ErrorResponse(`Please add a Url to shorten`, 400));
  }
  // Check videoUrl is valid
  if (!validUrl.isUri(videoUrl)) {
    return next(new ErrorResponse(`Invalid URL`, 400));
  }

  // Validate YouTubeURL
  ytdl.validateURL(videoUrl);

  // Get info from Video
  const video = await ytdl.getInfo(videoUrl);

  // Find ONLY video NO AUDIO formats to choose from
  const videoFormats = ytdl.filterFormats(video.formats, 'videoonly');

  // Find the highest ONLY video NO AUDIO to download
  const highQualityOnlyVideo = ytdl.chooseFormat(videoFormats, {
    quality: 'highest'
  });

  // Find ONY audio NO VIDEO formats to choose from
  const audioFormats = ytdl.filterFormats(video.formats, 'audioonly');

  // Find the highest ONLY audio NO VIDEO to download
  const highQualityOnlyAudio = ytdl.chooseFormat(audioFormats, {
    quality: 'highest'
  });

  // Find the video to download
  const finalVideo = ytdl.chooseFormat(video.formats, {
    quality: 'highest'
  });

  // Create document and store it to MongoDB
  const youtube = await YouTube.create({
    videoId: video.videoDetails.videoId,
    title: video.videoDetails.title,
    text: video.videoDetails.description,
    videoFetchedUrl: video.videoDetails.video_url,
    videoEmbedUrl: video.videoDetails.embed?.iframeUrl,
    videoToDownload: {
      url: finalVideo.url
    },
    videoOnly: {
      url: highQualityOnlyVideo.url
    },
    audioOnly: {
      url: highQualityOnlyAudio.url
    },
    thumbnails: video.videoDetails.thumbnails.map((image) => image.url),
    author: {
      name: video.videoDetails.author?.name,
      link: video.videoDetails.author?.user_url,
      channelUrl: video.videoDetails.author?.channel_url
    },
    category: video.videoDetails.category,
    likes: video.videoDetails?.likes,
    dislikes: video.videoDetails?.dislikes,
    keywords: video.videoDetails.keywords
      ? video.videoDetails.keywords
          .map((keyword) =>
            keyword
              .trim()
              .replace(/ +/g, ' ')
              .split(' ')
              .join('-')
              .toLowerCase()
          )
          .filter((keyword) => keyword.length !== 0)
      : 'no-tags',
    views: video.videoDetails?.viewCount
  });

  res.status(200).json({
    success: true,
    data: youtube
  });
});

// @desc    Delete all youtube links
// @route   DELETE /api/v1/extras/youtube/deleteall
// @access  Private
// @status  DONE
exports.deleteAllYoutube = asyncHandler(async (req, res, next) => {
  await YouTube.deleteMany();
  res.status(200).json({ success: true, data: {} });
});

Model:

const mongoose = require('mongoose');
const slugify = require('slugify');

const YouTubeSchema = new mongoose.Schema(
  {
    videoId: {
      type: String,
      trim: true,
      required: [true, 'Please add a name'],
      default: '0'
    },
    title: {
      type: String,
      trim: true,
      required: [true, 'Please add a name'],
      default: 'Unknown'
    },
    slug: {
      type: String,
      required: false
    },
    text: {
      type: String,
      trim: true,
      required: [true, 'Please add a name'],
      default: 'Unknown'
    },
    videoFetchedUrl: {
      type: String,
      trim: true,
      required: [true, 'Please add videoFetched url'],
      default: '...'
    },
    videoEmbedUrl: {
      type: String,
      trim: true,
      required: [true, 'Please add videoFetched url'],
      default: '...'
    },
    videoToDownload: {
      url: {
        type: String,
        trim: true,
        required: [true, 'Please add videoToDownload url'],
        default: '...'
      }
    },
    videoOnly: {
      url: {
        type: String,
        trim: true,
        required: [true, 'Please add videoOnly url'],
        default: '...'
      }
    },
    audioOnly: {
      url: {
        type: String,
        trim: true,
        required: [true, 'Please add an audioOnly url'],
        default: '...'
      }
    },
    thumbnails: {
      type: [String],
      trim: true,
      required: [true, 'Please add an audioOnly url'],
      default: '...'
    },
    author: {
      name: {
        type: String,
        trim: true,
        required: [true, 'Please add author'],
        default: 'Unknown'
      },
      link: {
        type: String,
        trim: true,
        required: [true, 'Please add author link'],
        default: 'Unknown'
      },
      channelUrl: {
        type: String,
        trim: true,
        required: [true, 'Please add channel url'],
        default: 'Unknown'
      }
    },
    category: {
      type: String,
      trim: true,
      required: [true, 'Please add category'],
      default: 'Uncategorized'
    },
    likes: {
      type: Number,
      required: [false, 'Please display likes'],
      default: 0
    },
    dislikes: {
      type: Number,
      required: [false, 'Please add dislikes'],
      default: 0
    },
    keywords: {
      type: [String],
      required: [true, 'Please add a term id option'],
      default: 'no-tags'
    },
    views: {
      type: Number,
      required: [false, 'Please add views'],
      default: 0
    }
  },
  {
    toJSON: { virtuals: true },
    toObject: { virtuals: true },
    timestamps: true,
    id: false
  }
);

// Create slug
YouTubeSchema.pre('save', function (next) {
  this.slug = slugify(this.title, { lower: true });
  next();
});

module.exports = mongoose.model('YouTube', YouTubeSchema);

On Another Note

If you would like to delete the data every certain hour, day, month, year, etc; you can simply create cron-job by installing node-cron:

npm i node-cron

Setting it up is simple, call the library and schedule the cron-jon according to your needs in this way:

const cron = require('node-cron')
const { deleteAllYoutube } = require('../controllers/extras/youtube')

const cronJobs = async (req, res, next) => {
  /*
   *
   * ON THE 15 OF EACH MONTH
   *
   */
  cron.schedule(
    '* * 15 * *',
    async () => {
      await deleteAllYoutube()
    }
  )
}

module.exports = cronJobs

I would highly recommend to create a file and name it as cronJob.js. You can have your cron-jobs logic separated from your server file and instead simply call the function in the server file:

/*
 *
 * SERVER/INDEX.js file
 *
 */

/*

 MORE CODE ABOVE

*/
const cronJobs = require('./YOUR_FOLDER/cronJobs')

// Make all cron-jobs available in your server file.
cronJobs()

/*

MORE CODE BELOW

*/

Don’t forget to check my previous articles on my blog and check this project on my portfolio age!.

Bye bye 🙂

Leave a Reply

Back to Top