magic_google

1.What's magic_google

This is an easy Google Searching crawler that you can get anything you want in the page by using it.

During the process of crawling,you need to pay attention to the limitation from google towards ip address and the warning of exception , so I suggest that you should pause running the program and own the Proxy ip

php - MagicGoogle

2.How to Use?

Run

pip install magic_google
# Or
pip install git+https://github.com/howie6879/magic_google.git
# Or
git clone https://github.com/howie6879/magic_google.git
cd magic_google
vim google_search.py
# Or 
python setup.py install

Example

from magic_google import MagicGoogle
import pprint

# Or PROXIES = None
PROXIES = [{
    'http': 'http://192.168.2.207:1080',
    'https': 'http://192.168.2.207:1080'
}]

# Or MagicGoogle()
mg = MagicGoogle(PROXIES)

#  Crawling the whole page
result = mg.search_page(query='python')

# Crawling url
for url in mg.search_url(query='python'):
    pprint.pprint(url)
    
# Output
# 'https://www.python.org/'
# 'https://www.python.org/downloads/'
# 'https://www.python.org/about/gettingstarted/'
# 'https://docs.python.org/2/tutorial/'
# 'https://docs.python.org/'
# 'https://en.wikipedia.org/wiki/Python_(programming_language)'
# 'https://www.codecademy.com/courses/introduction-to-python-6WeG3/0?curriculum_id=4f89dab3d788890003000096'
# 'https://www.codecademy.com/learn/python'
# 'https://developers.google.com/edu/python/'
# 'https://learnpythonthehardway.org/book/'
# 'https://www.continuum.io/downloads'

# Get {'title','url','text'}
for i in mg.search(query='python', num=1):
    pprint.pprint(i)
    
# Output
# {'text': 'The official home of the Python Programming Language.',
# 'title': 'Welcome to Python .org',
# 'url': 'https://www.python.org/'}

You can see google_search.py

If you need a big amount of querie but only having an ip address,I suggest you can have a time lapse between 5s ~ 30s.

The reason that it always return empty might be as follows:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://ipv4.google.com/sorry/index?continue=https://www.google.me/s****">here</A>.
</BODY></HTML>

Project Name	Stars	Repos Using This	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Wooyun_public	3,701			4 years ago			29		PHP
This repo is archived. Thanks for wooyun! 乌云公开漏洞、知识库爬虫和搜索 crawl and search for wooyun.org public bug(vulnerability) and drops
Ambar	1,797			2 years ago			2	mit	JavaScript
:mag: Ambar: Document Search Engine
Open Source Search Engine	1,376			7 months ago			89	apache-2.0	C++
Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.
Spider	919			5 months ago			88		Python
Python website crawler.
Angrysearch	866			a year ago			37	gpl-2.0	Python
Linux file search, instant results as you type
Ipfs Search	758		2	2 months ago	19	April 20, 2021	41	agpl-3.0	Go
Search engine for the Interplanetary Filesystem.
Fess	714	12	19	21 hours ago	135	June 13, 2022	27	apache-2.0	Java
Fess is very powerful and easily deployable Enterprise Search Server.
Tweetscraper	698			2 years ago	8	April 29, 2018	1	gpl-2.0	Python
TweetScraper is a simple crawler/spider for Twitter Search without using API
Go Dork	677			6 months ago	4	April 03, 2021	4	mit	Go
The fastest dork scanner written in Go.
Filemasta	625			a year ago			4	gpl-3.0	C#
A search application to explore, discover and share online files

Magic_google

magic_google

1.What's magic_google

2.How to Use?