AI Reads Books Page by Page
AI reads PDF page by page to build a knowledge base
Loading…
Description
#PDF Reading #AI Reading #Knowledge Base #Page-by-Page Analysis #Markdown #JSON #Long Document Processing #GitHub Project #AI Summary
This is an open-source script that focuses on allowing AI to read PDF page by page while building a knowledge base. It has already garnered about 2.3K Stars on GitHub. Instead of trying to shove hundreds of pages of PDF into the model at once, it adopts a more prudent approach: reading one page at a time and extracting knowledge page by page, continuously accumulating the results.
For books with hundreds of pages or even longer, this approach is very practical. Because directly handing the entire PDF to AI can easily be limited by the context window, resulting in a generalized summary that compresses a lot of chapter details, cases, concepts, and interrelations. This script breaks long documents into continuous small tasks, allowing AI to process content in the actual page order.
Each time a page is processed, it extracts the corresponding knowledge points and continuously writes them into a JSON knowledge base. After processing a certain number of pages, it can also automatically generate interim summaries; once the entire book is read, it generates a complete summary based on the accumulated content and saves it as a Markdown file. In other words, it does not just aim for "give me a summary at the end," but completes knowledge extraction, knowledge accumulation, and summarization during the reading process.
Page-by-Page PDF Reading: It does not attempt to shove the entire book into context at once but processes it page by page in order, making it more suitable for PDFs that are hundreds of pages long or longer.
Page-by-Page Knowledge Extraction: Each time a page is read, the model organizes key information from the current page, gradually structuring the content that was originally scattered throughout the book.
Automatic JSON Knowledge Base Construction: The extracted knowledge does not just exist in temporary context but is continuously saved into a JSON file, which can be further processed, retrieved, or developed later.
Interim Summaries: It can automatically organize interim summaries every few pages, avoiding complete reliance on the model to remember the content of the previous hundreds of pages when reading later.
Generate Complete Summary: After all pages are processed, it can generate the final summary of the entire PDF based on the already accumulated knowledge base and save it in Markdown format.
Separate Settings for Processing and Summarization Models: The page-by-page extraction and final summary can specify different AI models, for example, using a less expensive model for processing a large number of pages and letting a more capable model complete the final summary.
Support for Resuming from Breakpoints: If the processing of a hundreds-page PDF is unexpectedly interrupted halfway, you can load the already generated knowledge base to continue execution without having to start over from the first page each time.
Support for Limiting Processed Page Numbers: When testing prompts, models, or output effects, you can only process the first few pages, and once you are satisfied with the results, you can formally run the entire book to reduce unnecessary token and time consumption.
Suitable for Long Materials: Besides eBooks, this page-by-page processing approach is also suitable for collections of papers, course materials, research reports, technical documents, and other large PDFs, transforming one-time "AI reading" into a continuously accumulating structured knowledge base.
This is an open-source script that focuses on allowing AI to read PDF page by page while building a knowledge base. It has already garnered about 2.3K Stars on GitHub. Instead of trying to shove hundreds of pages of PDF into the model at once, it adopts a more prudent approach: reading one page at a time and extracting knowledge page by page, continuously accumulating the results.
For books with hundreds of pages or even longer, this approach is very practical. Because directly handing the entire PDF to AI can easily be limited by the context window, resulting in a generalized summary that compresses a lot of chapter details, cases, concepts, and interrelations. This script breaks long documents into continuous small tasks, allowing AI to process content in the actual page order.
Each time a page is processed, it extracts the corresponding knowledge points and continuously writes them into a JSON knowledge base. After processing a certain number of pages, it can also automatically generate interim summaries; once the entire book is read, it generates a complete summary based on the accumulated content and saves it as a Markdown file. In other words, it does not just aim for "give me a summary at the end," but completes knowledge extraction, knowledge accumulation, and summarization during the reading process.
Software Features
Page-by-Page PDF Reading: It does not attempt to shove the entire book into context at once but processes it page by page in order, making it more suitable for PDFs that are hundreds of pages long or longer.
Page-by-Page Knowledge Extraction: Each time a page is read, the model organizes key information from the current page, gradually structuring the content that was originally scattered throughout the book.
Automatic JSON Knowledge Base Construction: The extracted knowledge does not just exist in temporary context but is continuously saved into a JSON file, which can be further processed, retrieved, or developed later.
Interim Summaries: It can automatically organize interim summaries every few pages, avoiding complete reliance on the model to remember the content of the previous hundreds of pages when reading later.
Generate Complete Summary: After all pages are processed, it can generate the final summary of the entire PDF based on the already accumulated knowledge base and save it in Markdown format.
Separate Settings for Processing and Summarization Models: The page-by-page extraction and final summary can specify different AI models, for example, using a less expensive model for processing a large number of pages and letting a more capable model complete the final summary.
Support for Resuming from Breakpoints: If the processing of a hundreds-page PDF is unexpectedly interrupted halfway, you can load the already generated knowledge base to continue execution without having to start over from the first page each time.
Support for Limiting Processed Page Numbers: When testing prompts, models, or output effects, you can only process the first few pages, and once you are satisfied with the results, you can formally run the entire book to reduce unnecessary token and time consumption.
Suitable for Long Materials: Besides eBooks, this page-by-page processing approach is also suitable for collections of papers, course materials, research reports, technical documents, and other large PDFs, transforming one-time "AI reading" into a continuously accumulating structured knowledge base.
