|
此文章由 DDD888 原创或转贴,不代表本站立场和观点,版权归 oursteps.com.au 和作者 DDD888 所有!转贴必须注明作者、出处和本声明,并保持内容完整
本帖最后由 DDD888 于 2020-10-20 09:29 编辑
可以用golang来做爬虫和web server backend, reactjs做前台,mysql做数据库,家里连光纤,买个二手的服务器含32gb ram不要超过500新西兰元,配个商用ups,我就这样做的,就花点电费啦, 我把服务器放车库里
难易程度,对我专业开发软件26年,含23年新西兰工作经验的人来说piece of cake, so easy
当然啦,要速度快,爬虫可以用rust来写,我给个最简单的存网页的程序
extern crate reqwest;
use reqwest::Error;
use std::env;
use std::fs::File;
use std::io::Write;
use std: :Instant;
#[tokio::main]
async fn main() -> Result<(), Error> {
let args: Vec<String> = env::args().collect();
let args_len = args.len();
if !((args_len == 2) || (args_len == 3)) {
println!("Error missing url in command line");
return Ok(());
}
let request_url = &args[1];
let mut print_body = false;
if args_len == 3 {
if &args[2] == "true" {
print_body = true;
}
}
let start = Instant::now();
let client = reqwest::Client::builder()
.user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36")
.build()?;
let body = client.get(request_url).send().await?.text().await?;
let duration = start.elapsed();
if print_body {
//println!("{:?}", body);
let mut ofile = File::create("hello_world.txt").expect("unable to create file");
ofile.write_all(body.as_bytes()).expect("unable to write");
}
println!("Time elapsed is: {:?}", duration);
Ok(())
}
[dependencies]
tokio = { version = "0.2", features = ["macros", "tcp", "dns", "io-util"] }
reqwest = "0.10.8"
|
评分
-
查看全部评分
|