Alibaba Cloud DNS:Best practices for integrating the Alibaba Cloud HTTPDNS SDK in an iOS WebView

Overview

When an App WebView loads network requests, the iOS system can intercept these requests using system APIs and implement custom logic injection. However, after intercepting network requests in WebView, the iOS system must handle sending network requests based on IP addresses, receiving data, page redirection, page decoding, cookies, caching, and other functions. Overall, using SDK resolution for IP address-based direct connections in WebView has a high implementation threshold. Mobile operating systems provide limited support for this scenario and have several defects. Developers need strong code-level control over network/OS frameworks to avoid and optimize these issues.

UIWebView using NSURLProtocol to intercept requests

• Based on NSURLProtocol, you can intercept network requests sent by the upper-layer network library NSURLConnection/NSURLSession on the iOS system. Requests sent by UIWebView are also included.

• Register a custom NSURLProtocol through the interface to intercept UIWebView upper-layer network requests, create new network requests to handle data sending, receiving, redirection, and other processing logic, and return the results to the original request.

  [NSURLProtocol registerClass:[CustomProtocol class]];

• Overview of custom NSURLProtocol processing:

  • Intercept requests that need domain name resolution in canInitWithRequest.

  • After a request is intercepted, perform domain name resolution using Alibaba Cloud HTTPDNS.

  • After resolution completes, replace the URL.host field and HTTP Header Host domain like a normal request, and handle data sending, receiving, redirection, and other request processing.

  • Return the request processing results to the original UIWebView request through the NSURLProtocol interface.

• For NSURLProtocol usage reference, see Apple NSURLProtocol API. For Apple's official sample code, see Apple Sample Code - CustomHTTPProtocol.

WKWebView using private APIs for registration and interception of requests

WKWebView executes network requests in a process separate from the app process. Request data does not pass through the main process. Therefore, using NSURLProtocol directly on WKWebView cannot intercept requests.

Currently, there is no mature IP address-based direct connection solution for the WKWebView scenario. The following describes the method of using private APIs for registration and interception.

  // Register your own protocol
  [NSURLProtocol registerClass:[CustomProtocol class]];
  // Create WKWebview
  WKWebViewConfiguration * config = [[WKWebViewConfiguration alloc] init];
  WKWebView * wkWebView = [[WKWebView alloc] initWithFrame:CGRectMake(0, 0,   [UIScreen mainScreen].bounds.size.width, [UIScreen mainScreen].bounds.size.height) configuration:config];
  [wkWebView loadRequest:webViewReq];
  [self.view addSubview:wkWebView];
  //Register scheme
  Class cls = NSClassFromString(@"WKBrowsingContextController");
  SEL sel = NSSelectorFromString(@"registerSchemeForCustomProtocol:");
  if ([cls respondsToSelector:sel]) {
      // Through http and https requests, similarly through other Schemes but must satisfy ULR Loading System
      [cls performSelector:sel withObject:@"http"];
      [cls performSelector:sel withObject:@"https"];
  }

For NSURLProtocol usage reference, see Apple NSURLProtocol API. For Apple's official sample code, see Apple Sample Code - CustomHTTPProtocol.

Use the IP address-based direct connection solution for cookie processing in WKWebView

Before performing subsequent operations, take note of the following issues:

• WKWebView is not optimal for cookie management. Is cookie management optimized in WKWebView of iOS 11? If cookie management is optimized in WKWebView of iOS 11, how do users benefit from the optimization?

• When you use an IP address to connect to the server, the value of the Domain field in the cookies returned by the server is also an IP address. If the IP address is dynamic, the following issue may occur: The logon status verification of some HTML5 apps depends on cookies. However, requests in WKWebView do not automatically include cookies.

  WKWebView using NSURLProtocol to intercept requests cannot obtain cookie information

  iOS 11 introduced a new API WKHTTPCookieStore that can be used to intercept cookie information from WKWebView.

  Example:

  WKHTTPCookieStore *cookieStroe = self.webView.configuration.websiteDataStore.httpCookieStore;
     // get cookies
      [cookieStroe getAllCookies:^(NSArray<NSHTTPCookie *> * _Nonnull cookies) {
          NSLog(@"All cookies %@",cookies);
      }];
  
      // set cookie
      NSMutableDictionary *dict = [NSMutableDictionary dictionary];
      dict[NSHTTPCookieName] = @"userid";
      dict[NSHTTPCookieValue] = @"123";
      dict[NSHTTPCookieDomain] = @"xxxx.com";
      dict[NSHTTPCookiePath] = @"/";
  
      NSHTTPCookie *cookie = [NSHTTPCookie cookieWithProperties:dict];
      [cookieStroe setCookie:cookie completionHandler:^{
          NSLog(@"set cookie");
      }];
  
      // delete cookie
      [cookieStroe deleteCookie:cookie completionHandler:^{
          NSLog(@"delete cookie");
      }];

  Using iOS 11 API WKHTTPCookieStore to solve the problem of WKWebView not carrying cookies in the first request

• Issue description: The logon status verification of some HTML5 apps depends on cookies. However, requests in WKWebView do not automatically include cookies. For example, if you perform a logon operation at the Native layer, obtain cookie information, and store it locally using NSHTTPCookieStorage, the corresponding webpage is still in a logged-out state when you open it using WKWebView. If the logon operation is also performed in WebView, this issue will not occur.

• You can use the API of iOS 11 to resolve this issue. Each request sent to WKWebView includes cookies in WKHTTPCookieStore but may not include cookies in NSHTTPCookieStorage. Therefore, the first request sent to WKWebView does not include cookies.

• Troubleshooting methods:

  Before executing -[WKWebView loadRequest:], copy the content from NSHTTPCookieStorage to WKHTTPCookieStore to achieve WKWebView Cookie injection. The following sample code provides an example:

  [self copyNSHTTPCookieStorageToWKHTTPCookieStoreWithCompletionHandler:^{
              NSURL *url = [NSURL URLWithString:@"https://www.v2ex.com"];
              NSURLRequest *request = [NSURLRequest requestWithURL:url];
              [_webView loadRequest:request];
  }];

  - (void)copyNSHTTPCookieStorageToWKHTTPCookieStoreWithCompletionHandler:(nullable void (^)())theCompletionHandler; {
      NSArray *cookies = [[NSHTTPCookieStorage sharedHTTPCookieStorage] cookies];
      WKHTTPCookieStore *cookieStroe = self.webView.configuration.websiteDataStore.httpCookieStore;
      if (cookies.count == 0) {
          !theCompletionHandler ?: theCompletionHandler();
          return;
      }
      for (NSHTTPCookie *cookie in cookies) {
          [cookieStroe setCookie:cookie completionHandler:^{
              if ([[cookies lastObject] isEqual:cookie]) {
                  !theCompletionHandler ?: theCompletionHandler();
                  return;
              }
          }];
      }
  }

  Note

  This is an iOS 11 API. For systems before iOS 11, you need to handle it differently.

  Using APIs before iOS 11 to solve the problem of WKWebView not carrying cookies in the first request

  All WKWebView objects use the same WKProcessPool instance to allow WKWebView objects to share cookies. Cookies are classified into session cookies and persistent cookies. However, the WKProcessPool instance will be reset after the app process is killed and restarted, causing the loss of cookies and session cookie data in the WKProcessPool. Currently, it is not possible to save the WKProcessPool instance locally. You can use the method of putting cookies in the header.

   WKWebView * webView = [WKWebView new]; 
   NSMutableURLRequest * request = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:@"http://xxx.com/login"]]; 
   [request addValue:@"skey=skeyValue" forHTTPHeaderField:@"Cookie"]; 
   [webView loadRequest:request];

  The cookie value skey=skeyValue can also be obtained uniformly through the domain. The method of obtaining it can refer to the following utility class:

  ALIDNSCookieManager.h
  
  #ifndef ALIDNSCookieManager_h
  #define ALIDNSCookieManager_h
  
  // URL matching Cookie rule
  typedef BOOL (^ALIDNSCookieFilter)(NSHTTPCookie *, NSURL *);
  
  @interface ALIDNSCookieManager : NSObject
  
  + (instancetype)sharedInstance;
  
  /**
   Specify URL matching Cookie policy
  
   @param filter Filter
   */
  - (void)setCookieFilter:(ALIDNSCookieFilter)filter;
  
  /**
   Process and store cookies carried by HTTP Response
  
   @param headerFields HTTP Header Fields
   @param URL Find cookies associated with URL according to matching policy
   @return Return cookies added to storage
   */
  - (NSArray<NSHTTPCookie *> *)handleHeaderFields:(NSDictionary *)headerFields forURL:(NSURL *)URL;
  
  /**
   Match local Cookie storage, get request cookie string for corresponding URL
  
   @param URL Specify URL to find associated cookies according to matching policy
   @return Return request Cookie string for corresponding URL
   */
  - (NSString *)getRequestCookieHeaderForURL:(NSURL *)URL;
  
  /**
   Delete stored cookies
  
   @param URL Find URL associated cookies according to matching policy
   @return Return number of successfully deleted cookies
   */
  - (NSInteger)deleteCookieForURL:(NSURL *)URL;
  
  @end
  
  #endif /* ALIDNSCookieManager_h */
  
  ALIDNSCookieManager.m
  #import <Foundation/Foundation.h>
  #import "ALIDNSCookieManager.h"
  
  @implementation ALIDNSCookieManager
  {
      ALIDNSCookieFilter cookieFilter;
  }
  
  - (instancetype)init {
      if (self = [super init]) {
          /**
              The Cookie and URL matching policy set here is relatively simple, checking if URL.host contains the domain field of the Cookie
              You can call the setCookieFilter operation to specify the cookie matching rule,
              For example, you can set the domain field of Cookie to match the suffix of URL.host | Whether URL conforms to Cookie's path setting
              For detailed matching rules, refer to RFC 2965 Section 3.3
           */
          cookieFilter = ^BOOL(NSHTTPCookie *cookie, NSURL *URL) {
              if ([URL.host containsString:cookie.domain]) {
                  return YES;
              }
              return NO;
          };
      }
      return self;
  }
  
  + (instancetype)sharedInstance {
      static id singletonInstance = nil;
      static dispatch_once_t onceToken;
      dispatch_once(&onceToken, ^{
          if (!singletonInstance) {
              singletonInstance = [[super allocWithZone:NULL] init];
          }
      });
      return singletonInstance;
  }
  
  + (id)allocWithZone:(struct _NSZone *)zone {
      return [self sharedInstance];
  }
  
  - (id)copyWithZone:(struct _NSZone *)zone {
      return self;
  }
  
  - (void)setCookieFilter:(ALIDNSCookieFilter)filter {
      if (filter != nil) {
          cookieFilter = filter;
      }
  }
  
  - (NSArray<NSHTTPCookie *> *)handleHeaderFields:(NSDictionary *)headerFields forURL:(NSURL *)URL {
      NSArray *cookieArray = [NSHTTPCookie cookiesWithResponseHeaderFields:headerFields forURL:URL];
      if (cookieArray != nil) {
          NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage];
          for (NSHTTPCookie *cookie in cookieArray) {
              if (cookieFilter(cookie, URL)) {
                  NSLog(@"Add a cookie: %@", cookie);
                  [cookieStorage setCookie:cookie];
              }
          }
      }
      return cookieArray;
  }
  
  - (NSString *)getRequestCookieHeaderForURL:(NSURL *)URL {
      NSArray *cookieArray = [self searchAppropriateCookies:URL];
      if (cookieArray != nil && cookieArray.count > 0) {
          NSDictionary *cookieDic = [NSHTTPCookie requestHeaderFieldsWithCookies:cookieArray];
          if ([cookieDic objectForKey:@"Cookie"]) {
              return cookieDic[@"Cookie"];
          }
      }
      return nil;
  }
  
  - (NSArray *)searchAppropriateCookies:(NSURL *)URL {
      NSMutableArray *cookieArray = [NSMutableArray array];
      NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage];
      for (NSHTTPCookie *cookie in [cookieStorage cookies]) {
          if (cookieFilter(cookie, URL)) {
              NSLog(@"Search an appropriate cookie: %@", cookie);
              [cookieArray addObject:cookie];
          }
      }
      return cookieArray;
  }
  
  - (NSInteger)deleteCookieForURL:(NSURL *)URL {
      int delCount = 0;
      NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage];
      for (NSHTTPCookie *cookie in [cookieStorage cookies]) {
          if (cookieFilter(cookie, URL)) {
              NSLog(@"Delete a cookie: %@", cookie);
              [cookieStorage deleteCookie:cookie];
              delCount++;
          }
      }
      return delCount;
  }
  
  @end

  The following code provides an example of how to send requests:

  WKWebView * webView = [WKWebView new]; 
  NSMutableURLRequest * request = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:@"http://xxx.com/login"]]; 
  NSString *value = [[ALIDNSCookieManager sharedInstance] getRequestCookieHeaderForURL:url];
  [request setValue:value forHTTPHeaderField:@"Cookie"];
  [webView loadRequest:request];

  Example of how to receive and process a request:

   NSURLSessionTask *task = [session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
          if (!error) {
              NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)response;
              // Parse HTTP Response Header, store cookies
              [[ALIDNSCookieManager sharedInstance] handleHeaderFields:[httpResponse allHeaderFields] forURL:url];
          }
      }];
      [task resume];

  Use document.cookie to set cookies to solve cookie issues for subsequent page (same domain) Ajax and iframe requests.

  WKUserContentController* userContentController = [WKUserContentController new]; 
  WKUserScript * cookieScript = [[WKUserScript alloc] initWithSource: @"document.cookie = 'skey=skeyValue';" injectionTime:WKUserScriptInjectionTimeAtDocumentStart forMainFrameOnly:NO]; 
  [userContentController addUserScript:cookieScript];

  Cookie includes dynamic IP address causing logon failure

  If cookies are generated during the client logon session and multiple IP addresses are specified for the related domain name, cookies that correspond to the domain name are read when the domain name is used to access the client and cookies that correspond to an IP address are read when the IP address is used to access the client. In this case, if different IP addresses are specified for the domain name that you used to access the client two consecutive times, the logon session becomes invalid.

  If the WebView page in the app needs to use the logon session stored in system cookies, previously all local network requests in the app used domain names to access and could share the logon session in cookies. But now that local network requests use IP addresses after using HTTPDNS, WebView that still uses domain names to access cannot read the logon session stored in system cookies (system cookies correspond to IP addresses). After an IP address-based direct connection is established, the server returns cookies that include a dynamic IP address. As a result, the logon fails.

  After using IP addresses to access, the cookies returned by the server also use IP addresses. This may cause the inability to use local cookies when accessing with the corresponding domain name, or when accessing with different IP addresses belonging to the same domain name, the cookies do not match, causing logon failure.

  The proposed approach is as follows:

• You should intervene in cookie storage based on domain names.

• At the source, the API domain name returns a single IP address.

  The second method may cause DNS scheduling characteristics to be lost. This method is not recommended. The first method is more suitable.

  Using iOS 11 API WKHTTPCookieStore to solve WKWebView's cookie management problem

  Each time the server returns cookies, modify them before storage by replacing the IP address with the domain name. This way, local cookies are matched even if an IP address is used to access a web page each time a network request is received. This is because the value of the host field is manually changed to a domain name that corresponds to the IP address.

  After a network request is successful or a web page is loaded, you can change the IP address in the Domain field in cookies to a domain name. Sample code:

  - (void)updateWKHTTPCookieStoreDomainFromIP:(NSString *)IP toHost:(NSString *)host {
      WKHTTPCookieStore *cookieStroe = self.webView.configuration.websiteDataStore.httpCookieStore;
      [cookieStroe getAllCookies:^(NSArray<NSHTTPCookie *> * _Nonnull cookies) {
          [[cookies copy] enumerateObjectsUsingBlock:^(NSHTTPCookie * _Nonnull cookie, NSUInteger idx, BOOL * _Nonnull stop) {
              if ([cookie.domain isEqualToString:IP]) {
                  NSMutableDictionary<NSHTTPCookiePropertyKey, id> *dict = [NSMutableDictionary dictionaryWithDictionary:cookie.properties];
                  dict[NSHTTPCookieDomain] = host;
                  NSHTTPCookie *newCookie = [NSHTTPCookie cookieWithProperties:[dict copy]];
                  [cookieStroe setCookie:newCookie completionHandler:^{
                      [self logCookies];
                      [cookieStroe deleteCookie:cookie
                              completionHandler:^{
                                  [self logCookies];
                              }];
                  }];
              }
          }];
      }];
  }

  iOS 11 also provides the following API to help you determine the period of time when data in cookies can be replaced:

  @protocol WKHTTPCookieStoreObserver <NSObject>
  @optional
  - (void)cookiesDidChangeInCookieStore:(WKHTTPCookieStore *)cookieStore;
  @end

  //WKHTTPCookieStore
  /*! @abstract Adds a WKHTTPCookieStoreObserver object with the cookie store.
   @param observer The observer object to add.
   @discussion The observer is not retained by the receiver. It is your responsibility
   to unregister the observer before it becomes invalid.
   */
  - (void)addObserver:(id<WKHTTPCookieStoreObserver>)observer;
  
  /*! @abstract Removes a WKHTTPCookieStoreObserver object from the cookie store.
   @param observer The observer to remove.
   */
  - (void)removeObserver:(id<WKHTTPCookieStoreObserver>)observer;

  Function:

  @interface WebViewController ()<WKHTTPCookieStoreObserver>
  - (void)viewDidLoad {
      [super viewDidLoad];
      [NSURLProtocol registerClass:[WebViewURLProtocol class]];
      NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage];
      [cookieStorage setCookieAcceptPolicy:NSHTTPCookieAcceptPolicyAlways];
      WKHTTPCookieStore *cookieStroe = self.webView.configuration.websiteDataStore.httpCookieStore;
      [cookieStroe addObserver:self];
  
      [self.view addSubview:self.webView];
      //... ...
  }
  
  #pragma mark -
  #pragma mark - WKHTTPCookieStoreObserver Delegate Method
  
  - (void)cookiesDidChangeInCookieStore:(WKHTTPCookieStore *)cookieStore {
      [self updateWKHTTPCookieStoreDomainFromIP:CYLIP toHost:CYLHOST];
  }

  The implementation of the -updateWKHTTPCookieStoreDomainFromIP method has been given above.

  This solution requires the client to maintain an IP→HOST mapping relationship and be able to perform reverse lookup from IP to HOST, which has a high maintenance cost. The following introduces a more general method, which is also the processing method before iOS 11:

  Processing method before iOS 11: Manually manage cookie storage after NSURLProtocal interception:

  Steps: Save the original URL to the Header when doing IP replacement:

  + (NSURLRequest *)canonicalRequestForRequest:(NSURLRequest *)request {
      NSMutableURLRequest *mutableReq = [request mutableCopy];
      NSString *originalUrl = mutableReq.URL.absoluteString;
      NSURL *url = [NSURL URLWithString:originalUrl];
      // Call an API operation to obtain the IP address.
      NSArray *array = [[DNSResolver share] getIpsByCacheWithDomain:domain andExpiredIPEnabled:YES];
      NSString *ip = array.firstObject;
      if (ip) {
          NSRange hostFirstRange = [originalUrl rangeOfString:url.host];
          if (NSNotFound != hostFirstRange.location) {
              NSString *newUrl = [originalUrl stringByReplacingCharactersInRange:hostFirstRange withString:ip];
              mutableReq.URL = [NSURL URLWithString:newUrl];
              [mutableReq setValue:url.host forHTTPHeaderField:@"host"];
              // Add originalUrl to save the original URL
              [mutableReq addValue:originalUrl forHTTPHeaderField:@"originalUrl"];
          }
      }
      NSURLRequest *postRequestIncludeBody = [mutableReq cyl_getPostRequestIncludeBody];
      return postRequestIncludeBody;
  }

  Then after getting the data, manually manage cookies:

  - (void)handleCookiesFromResponse:(NSURLResponse *)response {
      NSString *originalURLString = [self.request valueForHTTPHeaderField:@"originalUrl"];
      if ([response isKindOfClass:[NSHTTPURLResponse class]]) {
          NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)response;
          NSDictionary<NSString *, NSString *> *allHeaderFields = httpResponse.allHeaderFields;
          if (originalURLString && originalURLString.length > 0) {
              NSArray *cookies = [NSHTTPCookie cookiesWithResponseHeaderFields:allHeaderFields forURL: [[NSURL alloc] initWithString:originalURLString]];
              if (cookies && cookies.count > 0) {
                  NSURL *originalURL = [NSURL URLWithString:originalURLString];
                  [[NSHTTPCookieStorage sharedHTTPCookieStorage] setCookies:cookies forURL:originalURL mainDocumentURL:nil];
              }
          }
      }
  }
  
  - (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task willPerformHTTPRedirection:(NSHTTPURLResponse *)response newRequest:(NSURLRequest *)newRequest completionHandler:(void (^)(NSURLRequest *))completionHandler {
      NSString *location = response.allHeaderFields[@"Location"];
      NSURL *url = [[NSURL alloc] initWithString:location];
      NSMutableURLRequest *mRequest = [newRequest mutableCopy];
      mRequest.URL = url;
      if (location && location.length > 0) {
          if ([[newRequest.HTTPMethod lowercaseString] isEqualToString:@"post"]) {
              // Redirect POST to GET.
              mRequest.HTTPMethod = @"GET";
              mRequest.HTTPBody = nil;
          }
          [mRequest setValue:nil forHTTPHeaderField:@"host"];
          // Include the cookies in the request.
          [self handleCookiesFromResponse:response];
          [XXXURLProtocol removePropertyForKey:XXXURLProtocolHandledKey inRequest:mRequest];
          completionHandler(mRequest);
      } else{
         completionHandler(mRequest);
      }
  }

  Step 3: Before the request is sent, include the cookies in the request.

  + (void)handleCookieWithRequest:(NSMutableURLRequest *)request {
      NSString* originalURLString = [request valueForHTTPHeaderField:@"originalUrl"];
      if (!originalURLString || originalURLString.length == 0) {
          return;
      }
      NSArray *cookies = [[NSHTTPCookieStorage sharedHTTPCookieStorage] cookies];
      if (cookies && cookies.count >0) {
          NSDictionary *cookieHeaders = [NSHTTPCookie requestHeaderFieldsWithCookies:cookies];
          NSString *cookieString = [cookieHeaders objectForKey:@"Cookie"];
          [request addValue:cookieString forHTTPHeaderField:@"Cookie"];
      }
  }
  
  + (NSURLRequest *)canonicalRequestForRequest:(NSURLRequest *)request {
      NSMutableURLRequest *mutableReq = [request mutableCopy];
  //...
       [self handleCookieWithRequest:mutableReq];
      return [mutableReq copy];
  }

HTTP 302 request

The cookie solution mentioned above cannot solve the cookie problem of 302 requests. For example, if the first request is http://www.a.com, we solve the cookie problem of that request by including cookies in the request header. Then the page redirects to http://www.b.com with a 302 status code. At this point, the http://www.b.com request may not be able to access because it does not carry cookies. Of course, since the callback function is called before each page jump:

 - (void)webView:(WKWebView *)webView decidePolicyForNavigationAction:(WKNavigationAction *)navigationAction decisionHandler:(void (^)(WKNavigationActionPolicy))decisionHandler;

You can intercept 302 requests in this callback function, copy the request, include cookies in the request header, and reload the request. However, this method still cannot solve the cookie problem of cross-domain requests in page iframes, because -[WKWebView loadRequest:] is only suitable for loading mainFrame requests.