摘要:
Disclosed herein are method, systems and architectures for normalizing identifiers corresponding to resources using normalization rules that can be generalized for use with different resources. By way of a non-limiting example, an identifier can be a uniform resource locator (URL), and a normalization rule can be used to normalize URLs that correspond to different resources, e.g., content. A normalization rule can be generated by generalizing two or more normalization rules corresponding to different resources, such that a content determinative component is generalized. A normalization rule can be defined to include a context portion used to determine the rule's applicability to an identifier, and a transformation portion that identifies the transformations to be applied to an applicable identifier to yield a normalized form of the URL. A generalization of two or more normalization rules can include a normalization of one or both of the context and transformation portions.
摘要:
Techniques are disclosed for detecting web pages with duplicate content. In one embodiment, a set of shingles is computed for each page of a group of pages. An aggregate set of shingles is determined based on the sets of shingles computed for the group of pages. A first subset from the aggregate set of shingles is determined by selecting, from the aggregate set, shingles whose frequencies in the aggregate set exceed a specified threshold. A modified set of shingles is generated for each page of the group of pages by removing, from the set of shingles for that page, any shingle included in the first subset. One or more duplicate pages in the group of pages are determined based at least in part on the modified sets of shingles generated for the group of pages.
摘要:
Techniques are disclosed for detecting web pages with duplicate content. In one embodiment, a set of shingles is computed for each page of a group of pages. An aggregate set of shingles is determined based on the sets of shingles computed for the group of pages. A first subset from the aggregate set of shingles is determined by selecting, from the aggregate set, shingles whose frequencies in the aggregate set exceed a specified threshold. A modified set of shingles is generated for each page of the group of pages by removing, from the set of shingles for that page, any shingle included in the first subset. One or more duplicate pages in the group of pages are determined based at least in part on the modified sets of shingles generated for the group of pages.
摘要:
A method and apparatus are provided for identifying if two websites are co-owned. In one example, the method includes obtaining redirect URL (uniform resource locator) pairs from the Internet, constructing a training set using the redirect URL pairs, constructing a feature set based on the training set, and learning co-ownership decisions based on the feature set and the training set.
摘要:
Method, system, and programs for computing similarity. Input data is first received from one or more data sources and then analyzed to obtain an input feature vector that characterizes the input data. An index is then generated based on the input feature vector and is used to archive the input data, where the value of the index is computed based on an improved Johnson-Lindenstrass transformation (FJLT) process. With the improved FJLT process, first, the sign of each feature in the input feature vector is randomly flipped to obtain a flipped vector. A Hadamard transformation is then applied to the flipped vector to obtain a transformed vector. An inner product between the transformed vector and a sparse vector is then computed to obtain a base vector, based on which the value of the index is determined.
摘要:
A system and method for implementing a multi-step challenge and response test includes steps or acts of: using an input/output subsystem for presenting a series of challenges to a user that require said user to correctly solve each challenge before a next challenge is revealed to the user; receiving the user's response to each challenge; and submitting a last response in the series of challenges to a server for validation. The method further includes: using a processor device configured to perform for each challenge in the series of challenges: internally validating the response by comparing the user's response to a correct response; and using the user's response, decrypting the next challenge to reveal the next challenge; wherein the next challenge remains obfuscated until a previous challenge is correctly solved.
摘要:
Embodiments of the present inversion relate to a two-pass compression scheme that achieves compression performance on par with existing methods while admitting individual message decompression. These methods provide both storage savings and lower end-user latency. They preserve the advantages of standard text compression in exploiting short-range similarities in data, while introducing a second step to take advantage of long-range similarities often present in certain types of structured data, e.g. email archival files.
摘要:
An image CAPTCHA having one or more images, a challenge, and a correct answer to the challenge is constructed by selecting the one or more images from a plurality of candidate images based at least in part on each image's public information and private information. The private information of each of the images is accessible only to an entity responsible for constructing the CAPTCHA. Optionally, the one or more images are selected further based on the specific type of the CAPTCHA to be constructed.
摘要:
Embodiments of the present inversion relate to a two-pass compression scheme that achieves compression performance on par with existing methods while admitting individual message decompression. These methods provide both storage savings and lower end-user latency. They preserve the advantages of standard text compression in exploiting short-range similarities in data, while introducing a second step to take advantage of long-range similarities often present in certain types of structured data, e.g. email archival files.
摘要:
Method, system, and programs for computing similarity. Input data is first received from one or more data sources and then analyzed to obtain an input feature vector that characterizes the input data. An index is then generated based on the input feature vector and is used to archive the input data, where the value of the index is computed based on an improved Johnson-Lindenstrass transformation (FJLT) process. With the improved FJLT process, first, the sign of each feature in the input feature vector is randomly flipped to obtain a flipped vector. A Hadamard transformation is then applied to the flipped vector to obtain a transformed vector. An inner product between the transformed vector and a sparse vector is then computed to obtain a base vector, based on which the value of the index is determined.