Rebase + some code improvements + fix extraction of age-restricted videos + update clients version

Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
This commit is contained in:
FireMasterK
2021-07-29 03:25:09 +05:30
committed by TiA4f8R
parent 7753556e66
commit 2eeb0a3403
10 changed files with 366 additions and 275 deletions

View File

@@ -92,7 +92,7 @@ public class YoutubeMixPlaylistExtractorTest {
@Test
public void getPage() throws Exception {
final byte[] body = JsonWriter.string(prepareJsonBuilder(
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
NewPipe.getPreferredLocalization(), NewPipe.getPreferredContentCountry())
.value("videoId", VIDEO_ID)
.value("playlistId", "RD" + VIDEO_ID)
@@ -176,7 +176,7 @@ public class YoutubeMixPlaylistExtractorTest {
@Test
public void getPage() throws Exception {
final byte[] body = JsonWriter.string(prepareJsonBuilder(
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
NewPipe.getPreferredLocalization(), NewPipe.getPreferredContentCountry())
.value("videoId", VIDEO_ID)
.value("playlistId", "RD" + VIDEO_ID)
@@ -259,7 +259,7 @@ public class YoutubeMixPlaylistExtractorTest {
@Test
public void getPage() throws Exception {
final byte[] body = JsonWriter.string(prepareJsonBuilder(
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
NewPipe.getPreferredLocalization(), NewPipe.getPreferredContentCountry())
.value("videoId", VIDEO_ID)
.value("playlistId", "RDMM" + VIDEO_ID)
@@ -373,7 +373,7 @@ public class YoutubeMixPlaylistExtractorTest {
@Test
public void getPage() throws Exception {
final byte[] body = JsonWriter.string(prepareJsonBuilder(
final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(
NewPipe.getPreferredLocalization(), NewPipe.getPreferredContentCountry())
.value("videoId", VIDEO_ID_OF_CHANNEL)
.value("playlistId", "RDCM" + CHANNEL_ID)

View File

@@ -27,7 +27,7 @@ public class YoutubeParsingHelperTest {
@Test
public void testAreHardcodedClientVersionAndKeyValid() throws IOException, ExtractionException {
assertTrue("Hardcoded client version and key are not valid anymore",
YoutubeParsingHelper.areHardcodedClientVersionAndKeyValid().orElse(false));
YoutubeParsingHelper.areHardcodedClientVersionAndKeyValid());
}
@Test