R中的地理编码IP地址

Question

我编写了这个简短的代码，通过使用 freegeoip.net 自动对 IP 地址进行地理编码（默认情况下每小时 15,000 次查询；出色的服务！）：

> library(RCurl)
Loading required package: bitops
> ip.lst = 
c("193.198.38.10","91.93.52.105","134.76.194.180","46.183.103.8")
> q = do.call(rbind, lapply(ip.lst, function(x){ 
  try( data.frame(t(strsplit(getURI(paste0("freegeoip.net/csv/", x)), ",")[[1]]), stringsAsFactors = FALSE) ) 
}))
> names(q) = c("ip","country_code","country_name","region_code","region_name","city","zip_code","time_zone","latitude","longitude","metro_code")
> str(q)
'data.frame':   4 obs. of  11 variables:
$ ip          : chr  "193.198.38.10" "91.93.52.105" "134.76.194.180" "46.183.103.8"
$ country_code: chr  "HR" "TR" "DE" "DE"
$ country_name: chr  "Croatia" "Turkey" "Germany" "Germany"
$ region_code : chr  "" "06" "NI" ""
$ region_name : chr  "" "Ankara" "Lower Saxony" ""
$ city        : chr  "" "Ankara" "Gottingen" ""
$ zip_code    : chr  "" "06450" "37079" ""
$ time_zone   : chr  "Europe/Zagreb" "Europe/Istanbul" "Europe/Berlin" ""
$ latitude    : chr  "45.1667" "39.9230" "51.5333" "51.2993"
$ longitude   : chr  "15.5000" "32.8378" "9.9333" "9.4910"
$ metro_code  : chr  "0\r\n" "0\r\n" "0\r\n" "0\r\n"

在三行代码中，您可以获得所有 IP 的坐标，包括城市/国家代码。我想知道这是否可以并行化以使其运行得更快？否则，对超过 10,000 个 IP 进行地理编码可能需要数小时。

Answer 1

library(rgeolocate)

ip_lst = c("193.198.38.10", "91.93.52.105", "134.76.194.180", "46.183.103.8")

maxmind(ip_lst, "~/Data/GeoLite2-City.mmdb", 
        fields=c("country_code", "country_name", "region_name", "city_name", 
                 "timezone", "latitude", "longitude"))

##   country_code country_name            region_name  city_name        timezone latitude longitude
## 1           HR      Croatia                   <NA>       <NA>   Europe/Zagreb  45.1667   15.5000
## 2           TR       Turkey               Istanbul   Istanbul Europe/Istanbul  41.0186   28.9647
## 3           DE      Germany           Lower Saxony Bilshausen   Europe/Berlin  51.6167   10.1667
## 4           DE      Germany North Rhine-Westphalia     Aachen   Europe/Berlin  50.7787    6.1085

包中有获取必要数据文件的说明。您正在提取的某些字段非常不准确（比任何 geoip 供应商都愿意承认的更严重）。如果您确实需要那些不可用的，请提交issue，我们会添加它们。

Answer 2

我发现multidplyr 是一个用于进行并行服务器调用的出色软件包。这是我找到的最好的指南，我强烈建议您阅读整篇文章以更好地了解该软件包的工作原理：http://www.business-science.io/code-tools/2016/12/18/multidplyr.html

library("devtools")
devtools::install_github("hadley/multidplyr")
library(parallel)
library(multidplyr)
library(RCurl)
library(tidyverse)

# Convert your example into a function
get_ip <- function(ip) {
  do.call(rbind, lapply(ip, function(x) {
    try(data.frame(t(strsplit(getURI(
      paste0("freegeoip.net/csv/", x)
    ), ",")[[1]]), stringsAsFactors = FALSE))
  })) %>% nest(X1:X11)
}

# Made ip.lst into a Tibble to make it work better with dplyr
ip.lst =
  tibble(
    ip = c(
      "193.198.38.10",
      "91.93.52.105",
      "134.76.194.180",
      "46.183.103.8",
      "193.198.38.10",
      "91.93.52.105",
      "134.76.194.180",
      "46.183.103.8"
    )
  )

# Create a cluster based on how many cores your machine has
cl <- detectCores()
cluster <- create_cluster(cores = cl)

# Create a partitioned tibble
by_group  <- partition(ip.lst, cluster = cluster)

# Send libraries and the function get_ip() to each cluster
by_group %>%
  cluster_library("tidyverse") %>%
  cluster_library("RCurl") %>%
  cluster_assign_value("get_ip", get_ip)

# Send parallel requests to the website and parse the results
q <- by_group %>%
  do(get_ip(.$ip)) %>% 
  collect() %>% 
  unnest() %>% 
  tbl_df() %>% 
  select(-PARTITION_ID)

# Set names of the results
names(q) = c(
  "ip",
  "country_code",
  "country_name",
  "region_code",
  "region_name",
  "city",
  "zip_code",
  "time_zone",
  "latitude",
  "longitude",
  "metro_code"
)