Haskell API 客户端分页

Question

我正在尝试构建一个 Haskell 客户端来使用 RESTful JSON API。我想获取一个页面，然后从响应中获取“next_page”键，并将其提供给另一个 api 请求的查询参数。我想继续这样做，直到我浏览完所有项目。在 Haskell 中最干净的方法是什么？我现在正在使用显式递归，但我觉得必须有更好的方法，也许是使用 writer 或 state monad。

编辑：添加了我的代码。我知道我在滥用 fromJust。

data Post = Post {
          title  :: !Text,
          domain :: !Text,
          score  :: Int,
          url    :: !Text
} deriving (Show)

instance FromJSON Post where
  parseJSON (Object v) = do
    objectData <- v .: "data"
    title  <- objectData .: "title"
    domain <- objectData .: "domain"
    score  <- objectData .: "score"
    url    <- objectData .: "url"
    return $ Post title domain score url

data GetPostsResult = GetPostsResult {
  posts :: [Post],
  after :: Maybe Text
} deriving (Show)

instance FromJSON GetPostsResult where
  parseJSON (Object v) = do
    rootData   <- v        .:  "data"
    posts      <- rootData .:  "children"
    afterCode <- rootData .:? "after"
    return $ GetPostsResult posts afterCode

fetchPage:: Text -> IO (Maybe ([Post],Maybe Text))
fetchPage afterCode = do
  let url = "https://www.reddit.com/r/videos/top/.json?sort=top&t=day&after=" ++ unpack afterCode
  b <- get url
  let jsonBody = b ^. responseBody
  let postResponse = decode jsonBody :: Maybe GetPostsResult
  let pagePosts = posts <$>  postResponse
  let nextAfterCode = after $ fromJust postResponse
  if isNothing pagePosts then return Nothing else return (Just (fromJust pagePosts,nextAfterCode))

getPosts :: Text -> [Post] -> IO [Post]
getPosts x y = do
  p <- liftIO $ fetchPage x
  let posts = fst (fromJust p)
  let afterParam =  snd (fromJust p)
  case afterParam of
    Nothing ->  return []
    Just aff -> getPosts aff (posts ++ y)

main = do
  a <- getPosts "" []
  print a

Answer 1

您的方法确实很有吸引力，因为它很简单，但是它有一个缺点，Posts 的列表只有在整个分页链结束后才可用。

我建议使用像Pipe、Conduit 和朋友这样的流媒体库。优点是您可以通过使用相应流库提供的函数来流式传输结果并限制要检索的帖子数量。这是Pipe的示例，我添加了必要的导入并添加了postsP：

{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Data.Text (Text)
import qualified Data.Text as T
import Network.Wreq
import Data.Maybe
import Control.Lens
import Control.Monad.IO.Class
import qualified Pipes as P
import Data.Foldable (for_)

data Post = Post {
          title  :: !Text,
          domain :: !Text,
          score  :: Int,
          url    :: !Text
} deriving (Show)

instance FromJSON Post where
  parseJSON (Object v) = do
    objectData <- v .: "data"
    title  <- objectData .: "title"
    domain <- objectData .: "domain"
    score  <- objectData .: "score"
    url    <- objectData .: "url"
    return $ Post title domain score url

data GetPostsResult = GetPostsResult {
  posts :: [Post],
  after :: Maybe Text
} deriving (Show)

instance FromJSON GetPostsResult where
  parseJSON (Object v) = do
    rootData   <- v        .:  "data"
    posts      <- rootData .:  "children"
    afterCode <- rootData .:? "after"
    return $ GetPostsResult posts afterCode

fetchPage:: Text -> IO (Maybe ([Post],Maybe Text))
fetchPage afterCode = do
  let url = "https://www.reddit.com/r/videos/top/.json?sort=top&t=day&after=" ++ T.unpack afterCode
  b <- get url
  let jsonBody = b ^. responseBody
  let postResponse = decode jsonBody :: Maybe GetPostsResult
  let pagePosts = posts <$>  postResponse
  let nextAfterCode = after $ fromJust postResponse
  if isNothing pagePosts then return Nothing else return (Just (fromJust pagePosts,nextAfterCode))

getPosts :: Text -> [Post] -> IO [Post]
getPosts x y = do
  p <- liftIO $ fetchPage x
  let posts = fst (fromJust p)
  let afterParam =  snd (fromJust p)
  case afterParam of
    Nothing ->  return []
    Just aff -> getPosts aff (posts ++ y)

postsP :: Text -> P.Producer Post IO ()
postsP x = do
  p <- liftIO (fetchPage x)
  for_ p $ \(posts,afterParam) -> do
    P.each posts
    for_ afterParam postsP

main = P.runEffect $ P.for (postsP "") (liftIO . print)

Answer 2

我建议你需要一个延续单子。在延续中，您可以拥有组装页面的逻辑，将其交给延续的调用者，然后重复。当您调用“get_pages”函数时，您将返回第一页和一个采用新参数的新函数。

听起来你需要this answer，它展示了如何创建这样一个monad。如果您在函数中需要任何其他单子状态，请使其成为单子转换器。