View source for Scrapping: Save text setiap URL

==FITUR:==
* Input dari `keywords.txt`
* Cari tiap keyword di Google (ambil `top-N` URL)
* Kunjungi tiap URL dan ambil kontennya (judul + paragraf)
* Simpan semua ke `scraped_results.csv`

==Kebutuhan:==

pip install googlesearch-python requests beautifulsoup4

==SCRIPT FULL:==

 from googlesearch import search
 import requests
 from bs4 import BeautifulSoup
 import csv
 import time
 
 def load_keywords(filename):
     with open(filename, 'r', encoding='utf-8') as f:
         return [line.strip() for line in f if line.strip()]
 
 def get_page_content(url):
     try:
         headers = {
             "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
         }
         response = requests.get(url, headers=headers, timeout=10)
         soup = BeautifulSoup(response.content, 'html.parser') 
 
         # Ambil judul halaman
         title = soup.title.string if soup.title else 'No Title'
         
         # Ambil konten paragraf utama
         paragraphs = soup.find_all('p')
         text_content = ' '.join([p.get_text() for p in paragraphs[:5]])  # Batasi 5 paragraf pertama
         return title.strip(), text_content.strip()
 
     except Exception as e:
         return 'Error', f"Failed to fetch content: {e}"
 
 def google_scrape_with_content(keywords, num_results=5, output_file='scraped_results.csv'):
     with open(output_file, mode='w', newline='', encoding='utf-8') as file:
         writer = csv.writer(file)
         writer.writerow(['Keyword', 'Rank', 'Title', 'URL', 'Content']) 
 
         for keyword in keywords:
             print(f"\n🔍 Searching for: {keyword}")
             try:
                 results = search(keyword, num_results=num_results)
                 for i, url in enumerate(results):
                     print(f"  → Fetching: {url}")
                     title, content = get_page_content(url)
                     writer.writerow([keyword, i+1, title, url, content])
                     time.sleep(2)  # Delay biar aman
             except Exception as e:
                 print(f"❌ Error while searching '{keyword}': {e}") 
 
     print(f"\n✅ All results + content saved to '{output_file}'")
 
 # Main
 if __name__ == '__main__':
     keywords = load_keywords('keywords.txt')
     google_scrape_with_content(keywords, num_results=5)

==Output (`scraped_results.csv`):==

 | Keyword | Rank | Title | URL | Content |
 |--------|------|-------|-----|---------|
 | berita teknologi Indonesia | 1 | Judul dari halaman | https://... | Paragraf-paragraf pertama |
 | ... | ... | ... | ... | ... |

==Tips:==
* Jangan pakai `num_results > 10` kalau nggak pakai delay besar.
* Bisa diubah agar simpan ke `.txt` atau `.json` juga.
* Mau filter halaman yang '''bukan berita'''? Bisa ditambahkan regex atau `if "news" in url`.

==Pranala Menarik==

* [[Scrapping]]