API tools faq
paste
luca020400

uksm.patch

luca020400
May 6th, 2015
528
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Diff 185.72 KB | None | 0 0
raw download clone embed print report
  1. From ce18c63b57b28f1dcaed093028fbca38bf505f74 Mon Sep 17 00:00:00 2001
  2. From: luca020400 <[email protected]>
  3. Date: Wed, 6 May 2015 12:35:04 +0200
  4. Subject: [PATCH 1/2] msm8226: Add UltraKSM 0.1.2.3
  5.  
  6. Change-Id: I5d8ad55385fb951b0bf4455fca59e9c06169f830
  7. ---
  8. Documentation/vm/00-INDEX   |    2 +
  9.  Documentation/vm/uksm.txt   |   58 +
  10.  fs/exec.c                   |    9 +-
  11.  fs/proc/meminfo.c           |    6 +
  12.  include/linux/ksm.h         |   44 +-
  13.  include/linux/mm_types.h    |    3 +
  14.  include/linux/mmzone.h      |    5 +-
  15.  include/linux/sradix-tree.h |   77 +
  16.  include/linux/uksm.h        |  145 ++
  17.  kernel/fork.c               |    4 +-
  18.  lib/Makefile                |    2 +-
  19.  lib/sradix-tree.c           |  476 ++++
  20.  mm/Kconfig                  |   26 +
  21.  mm/Makefile                 |    3 +-
  22.  mm/memory.c                 |   59 +-
  23.  mm/mmap.c                   |   43 +-
  24.  mm/rmap.c                   |    4 +-
  25.  mm/uksm.c                   | 5623 +++++++++++++++++++++++++++++++++++++++++++
  26.  mm/vmstat.c                 |    3 +
  27.  19 files changed, 6556 insertions(+), 36 deletions(-)
  28.  create mode 100644 Documentation/vm/uksm.txt
  29.  create mode 100644 include/linux/sradix-tree.h
  30.  create mode 100644 include/linux/uksm.h
  31.  create mode 100644 lib/sradix-tree.c
  32.  create mode 100644 mm/uksm.c
  33.  
  34. diff --git a/Documentation/vm/00-INDEX b/Documentation/vm/00-INDEX
  35. index 5481c8b..7141876 100644
  36. --- a/Documentation/vm/00-INDEX
  37. +++ b/Documentation/vm/00-INDEX
  38. @@ -14,6 +14,8 @@ hwpoison.txt
  39.     - explains what hwpoison is
  40.  ksm.txt
  41.     - how to use the Kernel Samepage Merging feature.
  42. +uksm.txt
  43. +   - Introduction to Ultra KSM
  44.  locking
  45.     - info on how locking and synchronization is done in the Linux vm code.
  46.  map_hugetlb.c
  47. diff --git a/Documentation/vm/uksm.txt b/Documentation/vm/uksm.txt
  48. new file mode 100644
  49. index 0000000..08bd645
  50. --- /dev/null
  51. +++ b/Documentation/vm/uksm.txt
  52. @@ -0,0 +1,58 @@
  53. +The Ultra Kernel Samepage Merging feature
  54. +----------------------------------------------
  55. +/*
  56. + * Ultra KSM. Copyright (C) 2011-2012 Nai Xia
  57. + *
  58. + * This is an improvement upon KSM. Some basic data structures and routines
  59. + * are borrowed from ksm.c .
  60. + *
  61. + * Its new features:
  62. + * 1. Full system scan:
  63. + *      It automatically scans all user processes' anonymous VMAs. Kernel-user
  64. + *      interaction to submit a memory area to KSM is no longer needed.
  65. + *
  66. + * 2. Rich area detection:
  67. + *      It automatically detects rich areas containing abundant duplicated
  68. + *      pages based. Rich areas are given a full scan speed. Poor areas are
  69. + *      sampled at a reasonable speed with very low CPU consumption.
  70. + *
  71. + * 3. Ultra Per-page scan speed improvement:
  72. + *      A new hash algorithm is proposed. As a result, on a machine with
  73. + *      Core(TM)2 Quad Q9300 CPU in 32-bit mode and 800MHZ DDR2 main memory, it
  74. + *      can scan memory areas that does not contain duplicated pages at speed of
  75. + *      627MB/sec ~ 2445MB/sec and can merge duplicated areas at speed of
  76. + *      477MB/sec ~ 923MB/sec.
  77. + *
  78. + * 4. Thrashing area avoidance:
  79. + *      Thrashing area(an VMA that has frequent Ksm page break-out) can be
  80. + *      filtered out. My benchmark shows it's more efficient than KSM's per-page
  81. + *      hash value based volatile page detection.
  82. + *
  83. + *
  84. + * 5. Misc changes upon KSM:
  85. + *      * It has a fully x86-opitmized memcmp dedicated for 4-byte-aligned page
  86. + *        comparison. It's much faster than default C version on x86.
  87. + *      * rmap_item now has an struct *page member to loosely cache a
  88. + *        address-->page mapping, which reduces too much time-costly
  89. + *        follow_page().
  90. + *      * The VMA creation/exit procedures are hooked to let the Ultra KSM know.
  91. + *      * try_to_merge_two_pages() now can revert a pte if it fails. No break_
  92. + *        ksm is needed for this case.
  93. + *
  94. + * 6. Full Zero Page consideration(contributed by Figo Zhang)
  95. + *    Now uksmd consider full zero pages as special pages and merge them to an
  96. + *    special unswappable uksm zero page.
  97. + */
  98. +
  99. +ChangeLog:
  100. +
  101. +2012-05-05 The creation of this Doc
  102. +2012-05-08 UKSM 0.1.1.1 libc crash bug fix, api clean up, doc clean up.
  103. +2012-05-28 UKSM 0.1.1.2 bug fix release
  104. +2012-06-26 UKSM 0.1.2-beta1 first beta release for 0.1.2
  105. +2012-07-2  UKSM 0.1.2-beta2
  106. +2012-07-10 UKSM 0.1.2-beta3
  107. +2012-07-26 UKSM 0.1.2 Fine grained speed control, more scan optimization.
  108. +2012-10-13 UKSM 0.1.2.1 Bug fixes.
  109. +2012-12-31 UKSM 0.1.2.2 Minor bug fixes
  110. +2014-07-02 UKSM 0.1.2.3 Fix a " __this_cpu_read() in preemptible bug"
  111. diff --git a/fs/exec.c b/fs/exec.c
  112. index 0a2d6b1..bd8cfc9 100644
  113. --- a/fs/exec.c
  114. +++ b/fs/exec.c
  115. @@ -19,7 +19,7 @@
  116.   * current->executable is only used by the procfs.  This allows a dispatch
  117.   * table to check for several different types  of binary formats.  We keep
  118.   * trying until we recognize the file or we run out of supported binary
  119. - * formats.
  120. + * formats.
  121.   */
  122.  
  123.  #include <linux/slab.h>
  124. @@ -55,6 +55,7 @@
  125.  #include <linux/pipe_fs_i.h>
  126.  #include <linux/oom.h>
  127.  #include <linux/compat.h>
  128. +#include <linux/ksm.h>
  129.  
  130.  #include <asm/uaccess.h>
  131.  #include <asm/mmu_context.h>
  132. @@ -1176,7 +1177,7 @@ void setup_new_exec(struct linux_binprm * bprm)
  133.        group */
  134.  
  135.     current->self_exec_id++;
  136. -          
  137. +
  138.     flush_signal_handlers(current, 0);
  139.     flush_old_files(current->files);
  140.  }
  141. @@ -1286,8 +1287,8 @@ static int check_unsafe_exec(struct linux_binprm *bprm)
  142.     return res;
  143.  }
  144.  
  145. -/*
  146. - * Fill the binprm structure from the inode.
  147. +/*
  148. + * Fill the binprm structure from the inode.
  149.   * Check permissions, then read the first 128 (BINPRM_BUF_SIZE) bytes
  150.   *
  151.   * This may be called multiple times for binary chains (scripts for example).
  152. diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
  153. index 80e4645..33f9e9b 100644
  154. --- a/fs/proc/meminfo.c
  155. +++ b/fs/proc/meminfo.c
  156. @@ -87,6 +87,9 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
  157.         "SUnreclaim:     %8lu kB\n"
  158.         "KernelStack:    %8lu kB\n"
  159.         "PageTables:     %8lu kB\n"
  160. +#ifdef CONFIG_UKSM
  161. +       "KsmZeroPages:   %8lu kB\n"
  162. +#endif
  163.  #ifdef CONFIG_QUICKLIST
  164.         "Quicklists:     %8lu kB\n"
  165.  #endif
  166. @@ -146,6 +149,9 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
  167.         K(global_page_state(NR_SLAB_UNRECLAIMABLE)),
  168.         global_page_state(NR_KERNEL_STACK) * THREAD_SIZE / 1024,
  169.         K(global_page_state(NR_PAGETABLE)),
  170. +#ifdef CONFIG_UKSM
  171. +       K(global_page_state(NR_UKSM_ZERO_PAGES)),
  172. +#endif
  173.  #ifdef CONFIG_QUICKLIST
  174.         K(quicklist_total_size()),
  175.  #endif
  176. diff --git a/include/linux/ksm.h b/include/linux/ksm.h
  177. index 3319a69..f4edf33 100644
  178. --- a/include/linux/ksm.h
  179. +++ b/include/linux/ksm.h
  180. @@ -22,21 +22,6 @@ struct page *ksm_does_need_to_copy(struct page *page,
  181.  #ifdef CONFIG_KSM
  182.  int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
  183.         unsigned long end, int advice, unsigned long *vm_flags);
  184. -int __ksm_enter(struct mm_struct *mm);
  185. -void __ksm_exit(struct mm_struct *mm);
  186. -
  187. -static inline int ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
  188. -{
  189. -   if (test_bit(MMF_VM_MERGEABLE, &oldmm->flags))
  190. -       return __ksm_enter(mm);
  191. -   return 0;
  192. -}
  193. -
  194. -static inline void ksm_exit(struct mm_struct *mm)
  195. -{
  196. -   if (test_bit(MMF_VM_MERGEABLE, &mm->flags))
  197. -       __ksm_exit(mm);
  198. -}
  199.  
  200.  /*
  201.   * A KSM page is one of those write-protected "shared pages" or "merged pages"
  202. @@ -90,6 +75,33 @@ int rmap_walk_ksm(struct page *page, int (*rmap_one)(struct page *,
  203.           struct vm_area_struct *, unsigned long, void *), void *arg);
  204.  void ksm_migrate_page(struct page *newpage, struct page *oldpage);
  205.  
  206. +#ifdef CONFIG_KSM_LEGACY
  207. +int __ksm_enter(struct mm_struct *mm);
  208. +void __ksm_exit(struct mm_struct *mm);
  209. +static inline int ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
  210. +{
  211. +   if (test_bit(MMF_VM_MERGEABLE, &oldmm->flags))
  212. +       return __ksm_enter(mm);
  213. +   return 0;
  214. +}
  215. +
  216. +static inline void ksm_exit(struct mm_struct *mm)
  217. +{
  218. +   if (test_bit(MMF_VM_MERGEABLE, &mm->flags))
  219. +       __ksm_exit(mm);
  220. +}
  221. +
  222. +#elif defined(CONFIG_UKSM)
  223. +static inline int ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
  224. +{
  225. +   return 0;
  226. +}
  227. +
  228. +static inline void ksm_exit(struct mm_struct *mm)
  229. +{
  230. +}
  231. +#endif /* !CONFIG_UKSM */
  232. +
  233.  #else  /* !CONFIG_KSM */
  234.  
  235.  static inline int ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
  236. @@ -142,4 +154,6 @@ static inline void ksm_migrate_page(struct page *newpage, struct page *oldpage)
  237.  #endif /* CONFIG_MMU */
  238.  #endif /* !CONFIG_KSM */
  239.  
  240. +#include <linux/uksm.h>
  241. +
  242.  #endif /* __LINUX_KSM_H */
  243. diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
  244. index 696b7d8..1b53adad 100644
  245. --- a/include/linux/mm_types.h
  246. +++ b/include/linux/mm_types.h
  247. @@ -262,6 +262,9 @@ struct vm_area_struct {
  248.  #ifdef CONFIG_NUMA
  249.     struct mempolicy *vm_policy;    /* NUMA policy for the VMA */
  250.  #endif
  251. +#ifdef CONFIG_UKSM
  252. +   struct vma_slot *uksm_vma_slot;
  253. +#endif
  254.  };
  255.  
  256.  struct core_thread {
  257. diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
  258. index 4f64bf7..933da82 100644
  259. --- a/include/linux/mmzone.h
  260. +++ b/include/linux/mmzone.h
  261. @@ -151,6 +151,9 @@ enum zone_stat_item {
  262.  #endif
  263.     NR_ANON_TRANSPARENT_HUGEPAGES,
  264.     NR_FREE_CMA_PAGES,
  265. +#ifdef CONFIG_UKSM
  266. +   NR_UKSM_ZERO_PAGES,
  267. +#endif
  268.     NR_VM_ZONE_STAT_ITEMS };
  269.  
  270.  /*
  271. @@ -797,7 +800,7 @@ static inline int is_normal_idx(enum zone_type idx)
  272.  }
  273.  
  274.  /**
  275. - * is_highmem - helper function to quickly check if a struct zone is a
  276. + * is_highmem - helper function to quickly check if a struct zone is a
  277.   *              highmem zone or not.  This is an attempt to keep references
  278.   *              to ZONE_{DMA/NORMAL/HIGHMEM/etc} in general code to a minimum.
  279.   * @zone - pointer to struct zone variable
  280. diff --git a/include/linux/sradix-tree.h b/include/linux/sradix-tree.h
  281. new file mode 100644
  282. index 0000000..6780fdb
  283. --- /dev/null
  284. +++ b/include/linux/sradix-tree.h
  285. @@ -0,0 +1,77 @@
  286. +#ifndef _LINUX_SRADIX_TREE_H
  287. +#define _LINUX_SRADIX_TREE_H
  288. +
  289. +
  290. +#define INIT_SRADIX_TREE(root, mask)                   \
  291. +do {                                   \
  292. +   (root)->height = 0;                     \
  293. +   (root)->gfp_mask = (mask);                  \
  294. +   (root)->rnode = NULL;                       \
  295. +} while (0)
  296. +
  297. +#define ULONG_BITS (sizeof(unsigned long) * 8)
  298. +#define SRADIX_TREE_INDEX_BITS  (8 /* CHAR_BIT */ * sizeof(unsigned long))
  299. +//#define SRADIX_TREE_MAP_SHIFT    6
  300. +//#define SRADIX_TREE_MAP_SIZE (1UL << SRADIX_TREE_MAP_SHIFT)
  301. +//#define SRADIX_TREE_MAP_MASK (SRADIX_TREE_MAP_SIZE-1)
  302. +
  303. +struct sradix_tree_node {
  304. +   unsigned int    height;     /* Height from the bottom */
  305. +   unsigned int    count;     
  306. +   unsigned int    fulls;      /* Number of full sublevel trees */
  307. +   struct sradix_tree_node *parent;
  308. +   void *stores[0];
  309. +};
  310. +
  311. +/* A simple radix tree implementation */
  312. +struct sradix_tree_root {
  313. +        unsigned int            height;
  314. +        struct sradix_tree_node *rnode;
  315. +
  316. +   /* Where found to have available empty stores in its sublevels */
  317. +        struct sradix_tree_node *enter_node;
  318. +   unsigned int shift;
  319. +   unsigned int stores_size;
  320. +   unsigned int mask;
  321. +   unsigned long min;  /* The first hole index */
  322. +   unsigned long num;
  323. +   //unsigned long *height_to_maxindex;
  324. +
  325. +   /* How the node is allocated and freed. */
  326. +   struct sradix_tree_node *(*alloc)(void);
  327. +   void (*free)(struct sradix_tree_node *node);
  328. +
  329. +   /* When a new node is added and removed */
  330. +   void (*extend)(struct sradix_tree_node *parent, struct sradix_tree_node *child);
  331. +   void (*assign)(struct sradix_tree_node *node, unsigned index, void *item);
  332. +   void (*rm)(struct sradix_tree_node *node, unsigned offset);
  333. +};
  334. +
  335. +struct sradix_tree_path {
  336. +   struct sradix_tree_node *node;
  337. +   int offset;
  338. +};
  339. +
  340. +static inline
  341. +void init_sradix_tree_root(struct sradix_tree_root *root, unsigned long shift)
  342. +{
  343. +   root->height = 0;
  344. +   root->rnode = NULL;
  345. +   root->shift = shift;
  346. +   root->stores_size = 1UL << shift;
  347. +   root->mask = root->stores_size - 1;
  348. +}
  349. +
  350. +
  351. +extern void *sradix_tree_next(struct sradix_tree_root *root,
  352. +              struct sradix_tree_node *node, unsigned long index,
  353. +              int (*iter)(void *, unsigned long));
  354. +
  355. +extern int sradix_tree_enter(struct sradix_tree_root *root, void **item, int num);
  356. +
  357. +extern void sradix_tree_delete_from_leaf(struct sradix_tree_root *root,
  358. +           struct sradix_tree_node *node, unsigned long index);
  359. +
  360. +extern void *sradix_tree_lookup(struct sradix_tree_root *root, unsigned long index);
  361. +
  362. +#endif /* _LINUX_SRADIX_TREE_H */
  363. diff --git a/include/linux/uksm.h b/include/linux/uksm.h
  364. new file mode 100644
  365. index 0000000..361eee2
  366. --- /dev/null
  367. +++ b/include/linux/uksm.h
  368. @@ -0,0 +1,145 @@
  369. +#ifndef __LINUX_UKSM_H
  370. +#define __LINUX_UKSM_H
  371. +/*
  372. + * Memory merging support.
  373. + *
  374. + * This code enables dynamic sharing of identical pages found in different
  375. + * memory areas, even if they are not shared by fork().
  376. + */
  377. +
  378. +/* if !CONFIG_UKSM this file should not be compiled at all. */
  379. +#ifdef CONFIG_UKSM
  380. +
  381. +#include <linux/bitops.h>
  382. +#include <linux/mm.h>
  383. +#include <linux/pagemap.h>
  384. +#include <linux/rmap.h>
  385. +#include <linux/sched.h>
  386. +
  387. +extern unsigned long zero_pfn __read_mostly;
  388. +extern unsigned long uksm_zero_pfn __read_mostly;
  389. +extern struct page *empty_uksm_zero_page;
  390. +
  391. +/* must be done before linked to mm */
  392. +extern void uksm_vma_add_new(struct vm_area_struct *vma);
  393. +extern void uksm_remove_vma(struct vm_area_struct *vma);
  394. +
  395. +#define UKSM_SLOT_NEED_SORT    (1 << 0)
  396. +#define UKSM_SLOT_NEED_RERAND  (1 << 1)
  397. +#define UKSM_SLOT_SCANNED      (1 << 2) /* It's scanned in this round */
  398. +#define UKSM_SLOT_FUL_SCANNED  (1 << 3)
  399. +#define UKSM_SLOT_IN_UKSM  (1 << 4)
  400. +
  401. +struct vma_slot {
  402. +   struct sradix_tree_node *snode;
  403. +   unsigned long sindex;
  404. +
  405. +   struct list_head slot_list;
  406. +   unsigned long fully_scanned_round;
  407. +   unsigned long dedup_num;
  408. +   unsigned long pages_scanned;
  409. +   unsigned long last_scanned;
  410. +   unsigned long pages_to_scan;
  411. +   struct scan_rung *rung;
  412. +   struct page **rmap_list_pool;
  413. +   unsigned int *pool_counts;
  414. +   unsigned long pool_size;
  415. +   struct vm_area_struct *vma;
  416. +   struct mm_struct *mm;
  417. +   unsigned long ctime_j;
  418. +   unsigned long pages;
  419. +   unsigned long flags;
  420. +   unsigned long pages_cowed; /* pages cowed this round */
  421. +   unsigned long pages_merged; /* pages merged this round */
  422. +   unsigned long pages_bemerged;
  423. +
  424. +   /* when it has page merged in this eval round */
  425. +   struct list_head dedup_list;
  426. +};
  427. +
  428. +static inline void uksm_unmap_zero_page(pte_t pte)
  429. +{
  430. +   if (pte_pfn(pte) == uksm_zero_pfn)
  431. +       __dec_zone_page_state(empty_uksm_zero_page, NR_UKSM_ZERO_PAGES);
  432. +}
  433. +
  434. +static inline void uksm_map_zero_page(pte_t pte)
  435. +{
  436. +   if (pte_pfn(pte) == uksm_zero_pfn)
  437. +       __inc_zone_page_state(empty_uksm_zero_page, NR_UKSM_ZERO_PAGES);
  438. +}
  439. +
  440. +static inline void uksm_cow_page(struct vm_area_struct *vma, struct page *page)
  441. +{
  442. +   if (vma->uksm_vma_slot && PageKsm(page))
  443. +       vma->uksm_vma_slot->pages_cowed++;
  444. +}
  445. +
  446. +static inline void uksm_cow_pte(struct vm_area_struct *vma, pte_t pte)
  447. +{
  448. +   if (vma->uksm_vma_slot && pte_pfn(pte) == uksm_zero_pfn)
  449. +       vma->uksm_vma_slot->pages_cowed++;
  450. +}
  451. +
  452. +static inline int uksm_flags_can_scan(unsigned long vm_flags)
  453. +{
  454. +   return !(vm_flags & (VM_PFNMAP | VM_IO  | VM_DONTEXPAND |
  455. +                 VM_RESERVED  | VM_HUGETLB | VM_INSERTPAGE |
  456. +                 VM_NONLINEAR | VM_MIXEDMAP | VM_SAO |
  457. +                 VM_SHARED  | VM_MAYSHARE | VM_GROWSUP
  458. +                 | VM_GROWSDOWN));
  459. +}
  460. +
  461. +static inline void uksm_vm_flags_mod(unsigned long *vm_flags_p)
  462. +{
  463. +   if (uksm_flags_can_scan(*vm_flags_p))
  464. +       *vm_flags_p |= VM_MERGEABLE;
  465. +}
  466. +
  467. +/*
  468. + * Just a wrapper for BUG_ON for where ksm_zeropage must not be. TODO: it will
  469. + * be removed when uksm zero page patch is stable enough.
  470. + */
  471. +static inline void uksm_bugon_zeropage(pte_t pte)
  472. +{
  473. +   BUG_ON(pte_pfn(pte) == uksm_zero_pfn);
  474. +}
  475. +#else
  476. +static inline void uksm_vma_add_new(struct vm_area_struct *vma)
  477. +{
  478. +}
  479. +
  480. +static inline void uksm_remove_vma(struct vm_area_struct *vma)
  481. +{
  482. +}
  483. +
  484. +static inline void uksm_unmap_zero_page(pte_t pte)
  485. +{
  486. +}
  487. +
  488. +static inline void uksm_map_zero_page(pte_t pte)
  489. +{
  490. +}
  491. +
  492. +static inline void uksm_cow_page(struct vm_area_struct *vma, struct page *page)
  493. +{
  494. +}
  495. +
  496. +static inline void uksm_cow_pte(struct vm_area_struct *vma, pte_t pte)
  497. +{
  498. +}
  499. +
  500. +static inline int uksm_flags_can_scan(unsigned long vm_flags)
  501. +{
  502. +   return 0;
  503. +}
  504. +
  505. +static inline void uksm_vm_flags_mod(unsigned long *vm_flags_p)
  506. +{
  507. +}
  508. +
  509. +static inline void uksm_bugon_zeropage(pte_t pte)
  510. +{
  511. +}
  512. +#endif /* !CONFIG_UKSM */
  513. +#endif /* __LINUX_UKSM_H */
  514. diff --git a/kernel/fork.c b/kernel/fork.c
  515. index 361b06b..d91e43f 100644
  516. --- a/kernel/fork.c
  517. +++ b/kernel/fork.c
  518. @@ -378,7 +378,7 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
  519.                 goto fail_nomem;
  520.             charge = len;
  521.         }
  522. -       tmp = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL);
  523. +       tmp = kmem_cache_zalloc(vm_area_cachep, GFP_KERNEL);
  524.         if (!tmp)
  525.             goto fail_nomem;
  526.         *tmp = *mpnt;
  527. @@ -430,7 +430,7 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
  528.         __vma_link_rb(mm, tmp, rb_link, rb_parent);
  529.         rb_link = &tmp->vm_rb.rb_right;
  530.         rb_parent = &tmp->vm_rb;
  531. -
  532. +       uksm_vma_add_new(tmp);
  533.         mm->map_count++;
  534.         retval = copy_page_range(mm, oldmm, mpnt);
  535.  
  536. diff --git a/lib/Makefile b/lib/Makefile
  537. index 3aa4428..e9e8bbf 100644
  538. --- a/lib/Makefile
  539. +++ b/lib/Makefile
  540. @@ -8,7 +8,7 @@ KBUILD_CFLAGS = $(subst -pg,,$(ORIG_CFLAGS))
  541.  endif
  542.  
  543.  lib-y := ctype.o string.o vsprintf.o cmdline.o \
  544. -    rbtree.o radix-tree.o dump_stack.o timerqueue.o\
  545. +    rbtree.o radix-tree.o sradix-tree.o dump_stack.o timerqueue.o\
  546.      idr.o int_sqrt.o extable.o prio_tree.o \
  547.      sha1.o md5.o irq_regs.o reciprocal_div.o argv_split.o \
  548.      proportions.o prio_heap.o ratelimit.o show_mem.o \
  549. diff --git a/lib/sradix-tree.c b/lib/sradix-tree.c
  550. new file mode 100644
  551. index 0000000..8d06329
  552. --- /dev/null
  553. +++ b/lib/sradix-tree.c
  554. @@ -0,0 +1,476 @@
  555. +#include <linux/errno.h>
  556. +#include <linux/mm.h>
  557. +#include <linux/mman.h>
  558. +#include <linux/spinlock.h>
  559. +#include <linux/slab.h>
  560. +#include <linux/gcd.h>
  561. +#include <linux/sradix-tree.h>
  562. +
  563. +static inline int sradix_node_full(struct sradix_tree_root *root, struct sradix_tree_node *node)
  564. +{
  565. +   return node->fulls == root->stores_size ||
  566. +       (node->height == 1 && node->count == root->stores_size);
  567. +}
  568. +
  569. +/*
  570. + * Extend a sradix tree so it can store key @index.
  571. + */
  572. +static int sradix_tree_extend(struct sradix_tree_root *root, unsigned long index)
  573. +{
  574. +   struct sradix_tree_node *node;
  575. +   unsigned int height;
  576. +
  577. +   if (unlikely(root->rnode == NULL)) {
  578. +       if (!(node = root->alloc()))
  579. +           return -ENOMEM;
  580. +
  581. +       node->height = 1;
  582. +       root->rnode = node;
  583. +       root->height = 1;
  584. +   }
  585. +
  586. +   /* Figure out what the height should be.  */
  587. +   height = root->height;
  588. +   index >>= root->shift * height;
  589. +
  590. +   while (index) {
  591. +       index >>= root->shift;
  592. +       height++;
  593. +   }
  594. +
  595. +   while (height > root->height) {
  596. +       unsigned int newheight;
  597. +       if (!(node = root->alloc()))
  598. +           return -ENOMEM;
  599. +
  600. +       /* Increase the height.  */
  601. +       node->stores[0] = root->rnode;
  602. +       root->rnode->parent = node;
  603. +       if (root->extend)
  604. +           root->extend(node, root->rnode);
  605. +
  606. +       newheight = root->height + 1;
  607. +       node->height = newheight;
  608. +       node->count = 1;
  609. +       if (sradix_node_full(root, root->rnode))
  610. +           node->fulls = 1;
  611. +
  612. +       root->rnode = node;
  613. +       root->height = newheight;
  614. +   }
  615. +
  616. +   return 0;
  617. +}
  618. +
  619. +/*
  620. + * Search the next item from the current node, that is not NULL
  621. + * and can satify root->iter().
  622. + */
  623. +void *sradix_tree_next(struct sradix_tree_root *root,
  624. +              struct sradix_tree_node *node, unsigned long index,
  625. +              int (*iter)(void *item, unsigned long height))
  626. +{
  627. +   unsigned long offset;
  628. +   void *item;
  629. +
  630. +   if (unlikely(node == NULL)) {
  631. +       node = root->rnode;
  632. +       for (offset = 0; offset < root->stores_size; offset++) {
  633. +           item = node->stores[offset];
  634. +           if (item && (!iter || iter(item, node->height)))
  635. +               break;
  636. +       }
  637. +
  638. +       if (unlikely(offset >= root->stores_size))
  639. +           return NULL;
  640. +
  641. +       if (node->height == 1)
  642. +           return item;
  643. +       else
  644. +           goto go_down;
  645. +   }
  646. +
  647. +   while (node) {
  648. +       offset = (index & root->mask) + 1;                 
  649. +       for (;offset < root->stores_size; offset++) {
  650. +           item = node->stores[offset];
  651. +           if (item && (!iter || iter(item, node->height)))
  652. +               break;
  653. +       }
  654. +
  655. +       if (offset < root->stores_size)
  656. +           break;
  657. +
  658. +       node = node->parent;
  659. +       index >>= root->shift;
  660. +   }
  661. +
  662. +   if (!node)
  663. +       return NULL;
  664. +
  665. +   while (node->height > 1) {
  666. +go_down:
  667. +       node = item;
  668. +       for (offset = 0; offset < root->stores_size; offset++) {
  669. +           item = node->stores[offset];
  670. +           if (item && (!iter || iter(item, node->height)))
  671. +               break;
  672. +       }
  673. +
  674. +       if (unlikely(offset >= root->stores_size))
  675. +           return NULL;
  676. +   }
  677. +
  678. +   BUG_ON(offset > root->stores_size);
  679. +
  680. +   return item;
  681. +}
  682. +
  683. +/*
  684. + * Blindly insert the item to the tree. Typically, we reuse the
  685. + * first empty store item.
  686. + */
  687. +int sradix_tree_enter(struct sradix_tree_root *root, void **item, int num)
  688. +{
  689. +   unsigned long index;
  690. +   unsigned int height;
  691. +   struct sradix_tree_node *node, *tmp = NULL;
  692. +   int offset, offset_saved;
  693. +   void **store = NULL;
  694. +   int error, i, j, shift;
  695. +
  696. +go_on:
  697. +   index = root->min;
  698. +
  699. +   if (root->enter_node && !sradix_node_full(root, root->enter_node)) {
  700. +       node = root->enter_node;
  701. +       BUG_ON((index >> (root->shift * root->height)));
  702. +   } else {
  703. +       node = root->rnode;
  704. +       if (node == NULL || (index >> (root->shift * root->height))
  705. +           || sradix_node_full(root, node)) {
  706. +           error = sradix_tree_extend(root, index);
  707. +           if (error)
  708. +               return error;
  709. +
  710. +           node = root->rnode;
  711. +       }
  712. +   }
  713. +
  714. +
  715. +   height = node->height;
  716. +   shift = (height - 1) * root->shift;
  717. +   offset = (index >> shift) & root->mask;
  718. +   while (shift > 0) {
  719. +       offset_saved = offset;
  720. +       for (; offset < root->stores_size; offset++) {
  721. +           store = &node->stores[offset];
  722. +           tmp = *store;
  723. +
  724. +           if (!tmp || !sradix_node_full(root, tmp))
  725. +               break;
  726. +       }
  727. +       BUG_ON(offset >= root->stores_size);
  728. +
  729. +       if (offset != offset_saved) {
  730. +           index += (offset - offset_saved) << shift;
  731. +           index &= ~((1UL << shift) - 1);
  732. +       }
  733. +
  734. +       if (!tmp) {
  735. +           if (!(tmp = root->alloc()))
  736. +               return -ENOMEM;
  737. +
  738. +           tmp->height = shift / root->shift;
  739. +           *store = tmp;
  740. +           tmp->parent = node;
  741. +           node->count++;
  742. +//         if (root->extend)
  743. +//             root->extend(node, tmp);
  744. +       }
  745. +
  746. +       node = tmp;
  747. +       shift -= root->shift;
  748. +       offset = (index >> shift) & root->mask;
  749. +   }
  750. +
  751. +   BUG_ON(node->height != 1);
  752. +
  753. +
  754. +   store = &node->stores[offset];
  755. +   for (i = 0, j = 0;
  756. +         j < root->stores_size - node->count &&
  757. +         i < root->stores_size - offset && j < num; i++) {
  758. +       if (!store[i]) {
  759. +           store[i] = item[j];
  760. +           if (root->assign)
  761. +               root->assign(node, index + i, item[j]);
  762. +           j++;
  763. +       }
  764. +   }
  765. +
  766. +   node->count += j;
  767. +   root->num += j;
  768. +   num -= j;
  769. +
  770. +   while (sradix_node_full(root, node)) {
  771. +       node = node->parent;
  772. +       if (!node)
  773. +           break;
  774. +
  775. +       node->fulls++;
  776. +   }
  777. +
  778. +   if (unlikely(!node)) {
  779. +       /* All nodes are full */
  780. +       root->min = 1 << (root->height * root->shift);
  781. +       root->enter_node = NULL;
  782. +   } else {
  783. +       root->min = index + i - 1;
  784. +       root->min |= (1UL << (node->height - 1)) - 1;
  785. +       root->min++;
  786. +       root->enter_node = node;
  787. +   }
  788. +
  789. +   if (num) {
  790. +       item += j;
  791. +       goto go_on;
  792. +   }
  793. +
  794. +   return 0;
  795. +}
  796. +
  797. +
  798. +/**
  799. + * sradix_tree_shrink    -    shrink height of a sradix tree to minimal
  800. + *      @root      sradix tree root
  801. + *  
  802. + */
  803. +static inline void sradix_tree_shrink(struct sradix_tree_root *root)
  804. +{
  805. +   /* try to shrink tree height */
  806. +   while (root->height > 1) {
  807. +       struct sradix_tree_node *to_free = root->rnode;
  808. +
  809. +       /*
  810. +        * The candidate node has more than one child, or its child
  811. +        * is not at the leftmost store, we cannot shrink.
  812. +        */
  813. +       if (to_free->count != 1 || !to_free->stores[0])
  814. +           break;
  815. +
  816. +       root->rnode = to_free->stores[0];
  817. +       root->rnode->parent = NULL;
  818. +       root->height--;
  819. +       if (unlikely(root->enter_node == to_free)) {
  820. +           root->enter_node = NULL;
  821. +       }
  822. +       root->free(to_free);
  823. +   }
  824. +}
  825. +
  826. +/*
  827. + * Del the item on the known leaf node and index
  828. + */
  829. +void sradix_tree_delete_from_leaf(struct sradix_tree_root *root,
  830. +                 struct sradix_tree_node *node, unsigned long index)
  831. +{
  832. +   unsigned int offset;
  833. +   struct sradix_tree_node *start, *end;
  834. +
  835. +   BUG_ON(node->height != 1);
  836. +
  837. +   start = node;
  838. +   while (node && !(--node->count))
  839. +       node = node->parent;
  840. +
  841. +   end = node;
  842. +   if (!node) {
  843. +       root->rnode = NULL;
  844. +       root->height = 0;
  845. +       root->min = 0;
  846. +       root->num = 0;
  847. +       root->enter_node = NULL;
  848. +   } else {
  849. +       offset = (index >> (root->shift * (node->height - 1))) & root->mask;
  850. +       if (root->rm)
  851. +           root->rm(node, offset);
  852. +       node->stores[offset] = NULL;
  853. +       root->num--;
  854. +       if (root->min > index) {
  855. +           root->min = index;
  856. +           root->enter_node = node;
  857. +       }
  858. +   }
  859. +
  860. +   if (start != end) {
  861. +       do {
  862. +           node = start;
  863. +           start = start->parent;
  864. +           if (unlikely(root->enter_node == node))
  865. +               root->enter_node = end;
  866. +           root->free(node);
  867. +       } while (start != end);
  868. +
  869. +       /*
  870. +        * Note that shrink may free "end", so enter_node still need to
  871. +        * be checked inside.
  872. +        */
  873. +       sradix_tree_shrink(root);
  874. +   } else if (node->count == root->stores_size - 1) {
  875. +       /* It WAS a full leaf node. Update the ancestors */
  876. +       node = node->parent;
  877. +       while (node) {
  878. +           node->fulls--;
  879. +           if (node->fulls != root->stores_size - 1)
  880. +               break;
  881. +
  882. +           node = node->parent;
  883. +       }
  884. +   }
  885. +}
  886. +
  887. +void *sradix_tree_lookup(struct sradix_tree_root *root, unsigned long index)
  888. +{
  889. +   unsigned int height, offset;
  890. +   struct sradix_tree_node *node;
  891. +   int shift;
  892. +
  893. +   node = root->rnode;
  894. +   if (node == NULL || (index >> (root->shift * root->height)))
  895. +       return NULL;
  896. +
  897. +   height = root->height;
  898. +   shift = (height - 1) * root->shift;
  899. +
  900. +   do {
  901. +       offset = (index >> shift) & root->mask;
  902. +       node = node->stores[offset];
  903. +       if (!node)
  904. +           return NULL;
  905. +
  906. +       shift -= root->shift;
  907. +   } while (shift >= 0);
  908. +
  909. +   return node;
  910. +}
  911. +
  912. +/*
  913. + * Return the item if it exists, otherwise create it in place
  914. + * and return the created item.
  915. + */
  916. +void *sradix_tree_lookup_create(struct sradix_tree_root *root,
  917. +           unsigned long index, void *(*item_alloc)(void))
  918. +{
  919. +   unsigned int height, offset;
  920. +   struct sradix_tree_node *node, *tmp;
  921. +   void *item;
  922. +   int shift, error;
  923. +
  924. +   if (root->rnode == NULL || (index >> (root->shift * root->height))) {
  925. +       if (item_alloc) {
  926. +           error = sradix_tree_extend(root, index);
  927. +           if (error)
  928. +               return NULL;
  929. +       } else {
  930. +           return NULL;
  931. +       }
  932. +   }
  933. +
  934. +   node = root->rnode;
  935. +   height = root->height;
  936. +   shift = (height - 1) * root->shift;
  937. +
  938. +   do {
  939. +       offset = (index >> shift) & root->mask;
  940. +       if (!node->stores[offset]) {
  941. +           if (!(tmp = root->alloc()))
  942. +               return NULL;
  943. +
  944. +           tmp->height = shift / root->shift;
  945. +           node->stores[offset] = tmp;
  946. +           tmp->parent = node;
  947. +           node->count++;
  948. +           node = tmp;
  949. +       } else {
  950. +           node = node->stores[offset];
  951. +       }
  952. +
  953. +       shift -= root->shift;
  954. +   } while (shift > 0);
  955. +
  956. +   BUG_ON(node->height != 1);
  957. +   offset = index & root->mask;
  958. +   if (node->stores[offset]) {
  959. +       return node->stores[offset];
  960. +   } else if (item_alloc) {
  961. +       if (!(item = item_alloc()))
  962. +           return NULL;
  963. +
  964. +       node->stores[offset] = item;
  965. +
  966. +       /*
  967. +        * NOTE: we do NOT call root->assign here, since this item is
  968. +        * newly created by us having no meaning. Caller can call this
  969. +        * if it's necessary to do so.
  970. +        */
  971. +
  972. +       node->count++;
  973. +       root->num++;
  974. +
  975. +       while (sradix_node_full(root, node)) {
  976. +           node = node->parent;
  977. +           if (!node)
  978. +               break;
  979. +
  980. +           node->fulls++;
  981. +       }
  982. +
  983. +       if (unlikely(!node)) {
  984. +           /* All nodes are full */
  985. +           root->min = 1 << (root->height * root->shift);
  986. +       } else {
  987. +           if (root->min == index) {
  988. +               root->min |= (1UL << (node->height - 1)) - 1;
  989. +               root->min++;
  990. +               root->enter_node = node;
  991. +           }
  992. +       }
  993. +
  994. +       return item;
  995. +   } else {
  996. +       return NULL;
  997. +   }
  998. +
  999. +}
  1000. +
  1001. +int sradix_tree_delete(struct sradix_tree_root *root, unsigned long index)
  1002. +{
  1003. +   unsigned int height, offset;
  1004. +   struct sradix_tree_node *node;
  1005. +   int shift;
  1006. +
  1007. +   node = root->rnode;
  1008. +   if (node == NULL || (index >> (root->shift * root->height)))
  1009. +       return -ENOENT;
  1010. +
  1011. +   height = root->height;
  1012. +   shift = (height - 1) * root->shift;
  1013. +
  1014. +   do {
  1015. +       offset = (index >> shift) & root->mask;
  1016. +       node = node->stores[offset];
  1017. +       if (!node)
  1018. +           return -ENOENT;
  1019. +
  1020. +       shift -= root->shift;
  1021. +   } while (shift > 0);
  1022. +
  1023. +   offset = index & root->mask;
  1024. +   if (!node->stores[offset])
  1025. +       return -ENOENT;
  1026. +
  1027. +   sradix_tree_delete_from_leaf(root, node, index);
  1028. +
  1029. +   return 0;
  1030. +}
  1031. diff --git a/mm/Kconfig b/mm/Kconfig
  1032. index bbab5a6..67bfb5f 100644
  1033. --- a/mm/Kconfig
  1034. +++ b/mm/Kconfig
  1035. @@ -245,6 +245,32 @@ config KSM
  1036.       See Documentation/vm/ksm.txt for more information: KSM is inactive
  1037.       until a program has madvised that an area is MADV_MERGEABLE, and
  1038.       root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set).
  1039. +choice
  1040. +   prompt "Choose UKSM/KSM strategy"
  1041. +   default UKSM
  1042. +   depends on KSM
  1043. +   help
  1044. +     This option allows to select a UKSM/KSM stragety.
  1045. +
  1046. +config UKSM
  1047. +   bool "Ultra-KSM for page merging"
  1048. +   depends on KSM
  1049. +   help
  1050. +   UKSM is inspired by the Linux kernel project \u2014 KSM(Kernel Same
  1051. +   page Merging), but with a fundamentally rewritten core algorithm. With
  1052. +   an advanced algorithm, UKSM now can transparently scans all anonymously
  1053. +   mapped user space applications with an significantly improved scan speed
  1054. +   and CPU efficiency. Since KVM is friendly to KSM, KVM can also benefit from
  1055. +   UKSM. Now UKSM has its first stable release and first real world enterprise user.
  1056. +   For more information, please goto its project page.
  1057. +   (www.kerneldedup.org)
  1058. +
  1059. +config KSM_LEGACY
  1060. +   bool "Legacy KSM implementation"
  1061. +   depends on KSM
  1062. +   help
  1063. +   The legacy KSM implementation from Redhat.
  1064. +endchoice
  1065.  
  1066.  config DEFAULT_MMAP_MIN_ADDR
  1067.          int "Low address space to protect from user allocation"
  1068. diff --git a/mm/Makefile b/mm/Makefile
  1069. index ccecbf9..2056f9b 100644
  1070. --- a/mm/Makefile
  1071. +++ b/mm/Makefile
  1072. @@ -33,7 +33,8 @@ obj-$(CONFIG_SPARSEMEM)   += sparse.o
  1073.  obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o
  1074.  obj-$(CONFIG_SLOB) += slob.o
  1075.  obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o
  1076. -obj-$(CONFIG_KSM) += ksm.o
  1077. +obj-$(CONFIG_KSM_LEGACY) += ksm.o
  1078. +obj-$(CONFIG_UKSM) += uksm.o
  1079.  obj-$(CONFIG_PAGE_POISONING) += debug-pagealloc.o
  1080.  obj-$(CONFIG_SLAB) += slab.o
  1081.  obj-$(CONFIG_SLUB) += slub.o
  1082. diff --git a/mm/memory.c b/mm/memory.c
  1083. index 571c3c1..878b5d5 100644
  1084. --- a/mm/memory.c
  1085. +++ b/mm/memory.c
  1086. @@ -114,6 +114,37 @@ __setup("norandmaps", disable_randmaps);
  1087.  unsigned long zero_pfn __read_mostly;
  1088.  unsigned long highest_memmap_pfn __read_mostly;
  1089.  
  1090. +#ifdef CONFIG_UKSM
  1091. +unsigned long uksm_zero_pfn __read_mostly;
  1092. +struct page *empty_uksm_zero_page;
  1093. +
  1094. +static int __init setup_uksm_zero_page(void)
  1095. +{
  1096. +   unsigned long addr;
  1097. +   addr = __get_free_pages(GFP_KERNEL | __GFP_ZERO, 0);
  1098. +   if (!addr)
  1099. +       panic("Oh boy, that early out of memory?");
  1100. +
  1101. +   empty_uksm_zero_page = virt_to_page((void *) addr);
  1102. +   SetPageReserved(empty_uksm_zero_page);
  1103. +
  1104. +   uksm_zero_pfn = page_to_pfn(empty_uksm_zero_page);
  1105. +
  1106. +   return 0;
  1107. +}
  1108. +core_initcall(setup_uksm_zero_page);
  1109. +
  1110. +static inline int is_uksm_zero_pfn(unsigned long pfn)
  1111. +{
  1112. +   return pfn == uksm_zero_pfn;
  1113. +}
  1114. +#else
  1115. +static inline int is_uksm_zero_pfn(unsigned long pfn)
  1116. +{
  1117. +   return 0;
  1118. +}
  1119. +#endif
  1120. +
  1121.  /*
  1122.   * CONFIG_MMU architectures set up ZERO_PAGE in their paging_init()
  1123.   */
  1124. @@ -125,6 +156,7 @@ static int __init init_zero_pfn(void)
  1125.  core_initcall(init_zero_pfn);
  1126.  
  1127.  
  1128. +
  1129.  #if defined(SPLIT_RSS_COUNTING)
  1130.  
  1131.  void sync_mm_rss(struct mm_struct *mm)
  1132. @@ -726,8 +758,10 @@ static inline int is_cow_mapping(vm_flags_t flags)
  1133.  #ifndef is_zero_pfn
  1134.  static inline int is_zero_pfn(unsigned long pfn)
  1135.  {
  1136. -   return pfn == zero_pfn;
  1137. +   return (pfn == zero_pfn) || (is_uksm_zero_pfn(pfn));
  1138.  }
  1139. +#else
  1140. +#define is_zero_pfn(pfn)   (is_zero_pfn(pfn) || is_uksm_zero_pfn(pfn))
  1141.  #endif
  1142.  
  1143.  #ifndef my_zero_pfn
  1144. @@ -913,6 +947,11 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
  1145.             rss[MM_ANONPAGES]++;
  1146.         else
  1147.             rss[MM_FILEPAGES]++;
  1148. +
  1149. +       /* Should return NULL in vm_normal_page() */
  1150. +       uksm_bugon_zeropage(pte);
  1151. +   } else {
  1152. +       uksm_map_zero_page(pte);
  1153.     }
  1154.  
  1155.  out_set_pte:
  1156. @@ -1148,8 +1187,10 @@ again:
  1157.             ptent = ptep_get_and_clear_full(mm, addr, pte,
  1158.                             tlb->fullmm);
  1159.             tlb_remove_tlb_entry(tlb, pte, addr);
  1160. -           if (unlikely(!page))
  1161. +           if (unlikely(!page)) {
  1162. +               uksm_unmap_zero_page(ptent);
  1163.                 continue;
  1164. +           }
  1165.             if (unlikely(details) && details->nonlinear_vma
  1166.                 && linear_page_index(details->nonlinear_vma,
  1167.                         addr) != page->index)
  1168. @@ -1675,7 +1716,7 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
  1169.  
  1170.     VM_BUG_ON(!!pages != !!(gup_flags & FOLL_GET));
  1171.  
  1172. -   /*
  1173. +   /*
  1174.      * Require read or write permissions.
  1175.      * If FOLL_FORCE is set, we only require the "MAY" flags.
  1176.      */
  1177. @@ -1722,7 +1763,7 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
  1178.                 page = vm_normal_page(vma, start, *pte);
  1179.                 if (!page) {
  1180.                     if (!(gup_flags & FOLL_DUMP) &&
  1181. -                        is_zero_pfn(pte_pfn(*pte)))
  1182. +                       (is_zero_pfn(pte_pfn(*pte))))
  1183.                         page = pte_page(*pte);
  1184.                     else {
  1185.                         pte_unmap(pte);
  1186. @@ -2542,8 +2583,10 @@ static inline void cow_user_page(struct page *dst, struct page *src, unsigned lo
  1187.             clear_page(kaddr);
  1188.         kunmap_atomic(kaddr);
  1189.         flush_dcache_page(dst);
  1190. -   } else
  1191. +   } else {
  1192.         copy_user_highpage(dst, src, va, vma);
  1193. +       uksm_cow_page(vma, src);
  1194. +   }
  1195.  }
  1196.  
  1197.  /*
  1198. @@ -2741,6 +2784,7 @@ gotten:
  1199.         new_page = alloc_zeroed_user_highpage_movable(vma, address);
  1200.         if (!new_page)
  1201.             goto oom;
  1202. +       uksm_cow_pte(vma, orig_pte);
  1203.     } else {
  1204.         new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, address);
  1205.         if (!new_page)
  1206. @@ -2762,8 +2806,11 @@ gotten:
  1207.                 dec_mm_counter_fast(mm, MM_FILEPAGES);
  1208.                 inc_mm_counter_fast(mm, MM_ANONPAGES);
  1209.             }
  1210. -       } else
  1211. +           uksm_bugon_zeropage(orig_pte);
  1212. +       } else {
  1213. +           uksm_unmap_zero_page(orig_pte);
  1214.             inc_mm_counter_fast(mm, MM_ANONPAGES);
  1215. +       }
  1216.         flush_cache_page(vma, address, pte_pfn(orig_pte));
  1217.         entry = mk_pte(new_page, vma->vm_page_prot);
  1218.         entry = maybe_mkwrite(pte_mkdirty(entry), vma);
  1219. diff --git a/mm/mmap.c b/mm/mmap.c
  1220. index c832385..7fc96ef 100644
  1221. --- a/mm/mmap.c
  1222. +++ b/mm/mmap.c
  1223. @@ -30,6 +30,7 @@
  1224.  #include <linux/perf_event.h>
  1225.  #include <linux/audit.h>
  1226.  #include <linux/khugepaged.h>
  1227. +#include <linux/ksm.h>
  1228.  
  1229.  #include <asm/uaccess.h>
  1230.  #include <asm/cacheflush.h>
  1231. @@ -65,7 +66,7 @@ static void unmap_region(struct mm_struct *mm,
  1232.   * MAP_SHARED  r: (no) no  r: (yes) yes    r: (no) yes r: (no) yes
  1233.   *     w: (no) no  w: (no) no  w: (yes) yes    w: (no) no
  1234.   *     x: (no) no  x: (no) yes x: (no) yes x: (yes) yes
  1235. - *    
  1236. + *
  1237.   * MAP_PRIVATE r: (no) no  r: (yes) yes    r: (no) yes r: (no) yes
  1238.   *     w: (no) no  w: (no) no  w: (copy) copy  w: (no) no
  1239.   *     x: (no) no  x: (no) yes x: (no) yes x: (yes) yes
  1240. @@ -236,6 +237,7 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
  1241.             removed_exe_file_vma(vma->vm_mm);
  1242.     }
  1243.     mpol_put(vma_policy(vma));
  1244. +   uksm_remove_vma(vma);
  1245.     kmem_cache_free(vm_area_cachep, vma);
  1246.     return next;
  1247.  }
  1248. @@ -501,9 +503,16 @@ int vma_adjust(struct vm_area_struct *vma, unsigned long start,
  1249.     long adjust_next = 0;
  1250.     int remove_next = 0;
  1251.  
  1252. +/*
  1253. + * to avoid deadlock, ksm_remove_vma must be done before any spin_lock is
  1254. + * acquired
  1255. + */
  1256. +   uksm_remove_vma(vma);
  1257. +
  1258.     if (next && !insert) {
  1259.         struct vm_area_struct *exporter = NULL;
  1260.  
  1261. +       uksm_remove_vma(next);
  1262.         if (end >= next->vm_end) {
  1263.             /*
  1264.              * vma expands, overlapping all the next, and
  1265. @@ -579,10 +588,10 @@ again:            remove_next = 1 + (end > next->vm_end);
  1266.         if (adjust_next)
  1267.             vma_prio_tree_remove(next, root);
  1268.     }
  1269. -
  1270.     vma->vm_start = start;
  1271.     vma->vm_end = end;
  1272.     vma->vm_pgoff = pgoff;
  1273. +
  1274.     if (adjust_next) {
  1275.         next->vm_start += adjust_next << PAGE_SHIFT;
  1276.         next->vm_pgoff += adjust_next;
  1277. @@ -635,10 +644,15 @@ again:            remove_next = 1 + (end > next->vm_end);
  1278.          */
  1279.         if (remove_next == 2) {
  1280.             next = vma->vm_next;
  1281. +           uksm_remove_vma(next);
  1282.             goto again;
  1283.         }
  1284. +   } else {
  1285. +       if (next && !insert)
  1286. +           uksm_vma_add_new(next);
  1287.     }
  1288.  
  1289. +   uksm_vma_add_new(vma);
  1290.     validate_mm(mm);
  1291.  
  1292.     return 0;
  1293. @@ -1006,6 +1020,9 @@ static unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
  1294.     vm_flags = calc_vm_prot_bits(prot) | calc_vm_flag_bits(flags) |
  1295.             mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC;
  1296.  
  1297. +   /* If uksm is enabled, we add VM_MERGABLE to new VMAs. */
  1298. +   uksm_vm_flags_mod(&vm_flags);
  1299. +
  1300.     if (flags & MAP_LOCKED)
  1301.         if (!can_do_mlock())
  1302.             return -EPERM;
  1303. @@ -1357,6 +1374,7 @@ munmap_back:
  1304.  
  1305.     vma_link(mm, vma, prev, rb_link, rb_parent);
  1306.     file = vma->vm_file;
  1307. +   uksm_vma_add_new(vma);
  1308.  
  1309.     /* Once vma denies write, undo our temporary denial count */
  1310.     if (correct_wcount)
  1311. @@ -1383,6 +1401,7 @@ unmap_and_free_vma:
  1312.     unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end);
  1313.     charged = 0;
  1314.  free_vma:
  1315. +   uksm_remove_vma(vma);
  1316.     kmem_cache_free(vm_area_cachep, vma);
  1317.  unacct_error:
  1318.     if (charged)
  1319. @@ -1458,7 +1477,7 @@ full_search:
  1320.         addr = vma->vm_end;
  1321.     }
  1322.  }
  1323. -#endif
  1324. +#endif
  1325.  
  1326.  void arch_unmap_area(struct mm_struct *mm, unsigned long addr)
  1327.  {
  1328. @@ -2012,6 +2031,8 @@ static int __split_vma(struct mm_struct * mm, struct vm_area_struct * vma,
  1329.     else
  1330.         err = vma_adjust(vma, vma->vm_start, addr, vma->vm_pgoff, new);
  1331.  
  1332. +   uksm_vma_add_new(new);
  1333. +
  1334.     /* Success. */
  1335.     if (!err)
  1336.         return 0;
  1337. @@ -2185,6 +2206,7 @@ static unsigned long do_brk(unsigned long addr, unsigned long len)
  1338.         return error;
  1339.  
  1340.     flags = VM_DATA_DEFAULT_FLAGS | VM_ACCOUNT | mm->def_flags;
  1341. +   uksm_vm_flags_mod(&flags);
  1342.  
  1343.     error = get_unmapped_area(NULL, addr, len, 0, MAP_FIXED);
  1344.     if (error & ~PAGE_MASK)
  1345. @@ -2253,6 +2275,7 @@ static unsigned long do_brk(unsigned long addr, unsigned long len)
  1346.     vma->vm_flags = flags;
  1347.     vma->vm_page_prot = vm_get_page_prot(flags);
  1348.     vma_link(mm, vma, prev, rb_link, rb_parent);
  1349. +   uksm_vma_add_new(vma);
  1350.  out:
  1351.     perf_event_mmap(vma);
  1352.     mm->total_vm += len >> PAGE_SHIFT;
  1353. @@ -2285,6 +2308,12 @@ void exit_mmap(struct mm_struct *mm)
  1354.     /* mm's last user has gone, and its about to be pulled down */
  1355.     mmu_notifier_release(mm);
  1356.  
  1357. +   /*
  1358. +    * Taking write lock on mmap_sem does not harm others,
  1359. +    * but it's crucial for uksm to avoid races.
  1360. +    */
  1361. +   down_write(&mm->mmap_sem);
  1362. +
  1363.     if (mm->locked_vm) {
  1364.         vma = mm->mmap;
  1365.         while (vma) {
  1366. @@ -2318,6 +2347,11 @@ void exit_mmap(struct mm_struct *mm)
  1367.     while (vma)
  1368.         vma = remove_vma(vma);
  1369.  
  1370. +   mm->mmap = NULL;
  1371. +   mm->mm_rb = RB_ROOT;
  1372. +   mm->mmap_cache = NULL;
  1373. +   up_write(&mm->mmap_sem);
  1374. +
  1375.     BUG_ON(mm->nr_ptes > (FIRST_USER_ADDRESS+PMD_SIZE-1)>>PMD_SHIFT);
  1376.  }
  1377.  
  1378. @@ -2427,6 +2461,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
  1379.             if (new_vma->vm_ops && new_vma->vm_ops->open)
  1380.                 new_vma->vm_ops->open(new_vma);
  1381.             vma_link(mm, new_vma, prev, rb_link, rb_parent);
  1382. +           uksm_vma_add_new(new_vma);
  1383.         }
  1384.     }
  1385.     return new_vma;
  1386. @@ -2532,10 +2567,10 @@ int install_special_mapping(struct mm_struct *mm,
  1387.     ret = insert_vm_struct(mm, vma);
  1388.     if (ret)
  1389.         goto out;
  1390. -
  1391.     mm->total_vm += len >> PAGE_SHIFT;
  1392.  
  1393.     perf_event_mmap(vma);
  1394. +   uksm_vma_add_new(vma);
  1395.  
  1396.     return 0;
  1397.  
  1398. diff --git a/mm/rmap.c b/mm/rmap.c
  1399. index aa95e59..a3a2f57 100644
  1400. --- a/mm/rmap.c
  1401. +++ b/mm/rmap.c
  1402. @@ -1006,9 +1006,9 @@ void page_move_anon_rmap(struct page *page,
  1403.  
  1404.  /**
  1405.   * __page_set_anon_rmap - set up new anonymous rmap
  1406. - * @page:  Page to add to rmap
  1407. + * @page:  Page to add to rmap
  1408.   * @vma:   VM area to add page to.
  1409. - * @address:   User virtual address of the mapping
  1410. + * @address:   User virtual address of the mapping
  1411.   * @exclusive: the page is exclusively owned by the current process
  1412.   */
  1413.  static void __page_set_anon_rmap(struct page *page,
  1414. diff --git a/mm/uksm.c b/mm/uksm.c
  1415. new file mode 100644
  1416. index 0000000..8304d5b
  1417. --- /dev/null
  1418. +++ b/mm/uksm.c
  1419. @@ -0,0 +1,5623 @@
  1420. +/*
  1421. + * Ultra KSM. Copyright (C) 2011-2012 Nai Xia
  1422. + *
  1423. + * This is an improvement upon KSM. Some basic data structures and routines
  1424. + * are borrowed from ksm.c .
  1425. + *
  1426. + * Its new features:
  1427. + * 1. Full system scan:
  1428. + *      It automatically scans all user processes' anonymous VMAs. Kernel-user
  1429. + *      interaction to submit a memory area to KSM is no longer needed.
  1430. + *
  1431. + * 2. Rich area detection:
  1432. + *      It automatically detects rich areas containing abundant duplicated
  1433. + *      pages based. Rich areas are given a full scan speed. Poor areas are
  1434. + *      sampled at a reasonable speed with very low CPU consumption.
  1435. + *
  1436. + * 3. Ultra Per-page scan speed improvement:
  1437. + *      A new hash algorithm is proposed. As a result, on a machine with
  1438. + *      Core(TM)2 Quad Q9300 CPU in 32-bit mode and 800MHZ DDR2 main memory, it
  1439. + *      can scan memory areas that does not contain duplicated pages at speed of
  1440. + *      627MB/sec ~ 2445MB/sec and can merge duplicated areas at speed of
  1441. + *      477MB/sec ~ 923MB/sec.
  1442. + *
  1443. + * 4. Thrashing area avoidance:
  1444. + *      Thrashing area(an VMA that has frequent Ksm page break-out) can be
  1445. + *      filtered out. My benchmark shows it's more efficient than KSM's per-page
  1446. + *      hash value based volatile page detection.
  1447. + *
  1448. + *
  1449. + * 5. Misc changes upon KSM:
  1450. + *      * It has a fully x86-opitmized memcmp dedicated for 4-byte-aligned page
  1451. + *        comparison. It's much faster than default C version on x86.
  1452. + *      * rmap_item now has an struct *page member to loosely cache a
  1453. + *        address-->page mapping, which reduces too much time-costly
  1454. + *        follow_page().
  1455. + *      * The VMA creation/exit procedures are hooked to let the Ultra KSM know.
  1456. + *      * try_to_merge_two_pages() now can revert a pte if it fails. No break_
  1457. + *        ksm is needed for this case.
  1458. + *
  1459. + * 6. Full Zero Page consideration(contributed by Figo Zhang)
  1460. + *    Now uksmd consider full zero pages as special pages and merge them to an
  1461. + *    special unswappable uksm zero page.
  1462. + */
  1463. +
  1464. +#include <linux/errno.h>
  1465. +#include <linux/mm.h>
  1466. +#include <linux/fs.h>
  1467. +#include <linux/mman.h>
  1468. +#include <linux/sched.h>
  1469. +#include <linux/rwsem.h>
  1470. +#include <linux/pagemap.h>
  1471. +#include <linux/rmap.h>
  1472. +#include <linux/spinlock.h>
  1473. +#include <linux/jhash.h>
  1474. +#include <linux/delay.h>
  1475. +#include <linux/kthread.h>
  1476. +#include <linux/wait.h>
  1477. +#include <linux/slab.h>
  1478. +#include <linux/rbtree.h>
  1479. +#include <linux/memory.h>
  1480. +#include <linux/mmu_notifier.h>
  1481. +#include <linux/swap.h>
  1482. +#include <linux/ksm.h>
  1483. +#include <linux/crypto.h>
  1484. +#include <linux/scatterlist.h>
  1485. +#include <crypto/hash.h>
  1486. +#include <linux/random.h>
  1487. +#include <linux/math64.h>
  1488. +#include <linux/gcd.h>
  1489. +#include <linux/freezer.h>
  1490. +#include <linux/sradix-tree.h>
  1491. +
  1492. +#include <asm/tlbflush.h>
  1493. +#include "internal.h"
  1494. +
  1495. +#ifdef CONFIG_X86
  1496. +#undef memcmp
  1497. +
  1498. +#ifdef CONFIG_X86_32
  1499. +#define memcmp memcmpx86_32
  1500. +/*
  1501. + * Compare 4-byte-aligned address s1 and s2, with length n
  1502. + */
  1503. +int memcmpx86_32(void *s1, void *s2, size_t n)
  1504. +{
  1505. +   size_t num = n / 4;
  1506. +   register int res;
  1507. +
  1508. +   __asm__ __volatile__
  1509. +   (
  1510. +    "testl %3,%3\n\t"
  1511. +    "repe; cmpsd\n\t"
  1512. +    "je        1f\n\t"
  1513. +    "sbbl      %0,%0\n\t"
  1514. +    "orl       $1,%0\n"
  1515. +    "1:"
  1516. +    : "=&a" (res), "+&S" (s1), "+&D" (s2), "+&c" (num)
  1517. +    : "0" (0)
  1518. +    : "cc");
  1519. +
  1520. +   return res;
  1521. +}
  1522. +
  1523. +/*
  1524. + * Check the page is all zero ?
  1525. + */
  1526. +static int is_full_zero(const void *s1, size_t len)
  1527. +{
  1528. +   unsigned char same;
  1529. +
  1530. +   len /= 4;
  1531. +
  1532. +   __asm__ __volatile__
  1533. +   ("repe; scasl;"
  1534. +    "sete %0"
  1535. +    : "=qm" (same), "+D" (s1), "+c" (len)
  1536. +    : "a" (0)
  1537. +    : "cc");
  1538. +
  1539. +   return same;
  1540. +}
  1541. +
  1542. +
  1543. +#elif defined(CONFIG_X86_64)
  1544. +#define memcmp memcmpx86_64
  1545. +/*
  1546. + * Compare 8-byte-aligned address s1 and s2, with length n
  1547. + */
  1548. +int memcmpx86_64(void *s1, void *s2, size_t n)
  1549. +{
  1550. +   size_t num = n / 8;
  1551. +   register int res;
  1552. +
  1553. +   __asm__ __volatile__
  1554. +   (
  1555. +    "testq %q3,%q3\n\t"
  1556. +    "repe; cmpsq\n\t"
  1557. +    "je        1f\n\t"
  1558. +    "sbbq      %q0,%q0\n\t"
  1559. +    "orq       $1,%q0\n"
  1560. +    "1:"
  1561. +    : "=&a" (res), "+&S" (s1), "+&D" (s2), "+&c" (num)
  1562. +    : "0" (0)
  1563. +    : "cc");
  1564. +
  1565. +   return res;
  1566. +}
  1567. +
  1568. +static int is_full_zero(const void *s1, size_t len)
  1569. +{
  1570. +   unsigned char same;
  1571. +
  1572. +   len /= 8;
  1573. +
  1574. +   __asm__ __volatile__
  1575. +   ("repe; scasq;"
  1576. +    "sete %0"
  1577. +    : "=qm" (same), "+D" (s1), "+c" (len)
  1578. +    : "a" (0)
  1579. +    : "cc");
  1580. +
  1581. +   return same;
  1582. +}
  1583. +
  1584. +#endif
  1585. +#else
  1586. +static int is_full_zero(const void *s1, size_t len)
  1587. +{
  1588. +   const unsigned long *src = s1;
  1589. +   int i;
  1590. +
  1591. +   len /= sizeof(*src);
  1592. +
  1593. +   for (i = 0; i < len; i++) {
  1594. +       if (src[i])
  1595. +           return 0;
  1596. +   }
  1597. +
  1598. +   return 1;
  1599. +}
  1600. +#endif
  1601. +
  1602. +#define U64_MAX        (~((u64)0))
  1603. +#define UKSM_RUNG_ROUND_FINISHED  (1 << 0)
  1604. +#define TIME_RATIO_SCALE   10000
  1605. +
  1606. +#define SLOT_TREE_NODE_SHIFT   8
  1607. +#define SLOT_TREE_NODE_STORE_SIZE  (1UL << SLOT_TREE_NODE_SHIFT)
  1608. +struct slot_tree_node {
  1609. +   unsigned long size;
  1610. +   struct sradix_tree_node snode;
  1611. +   void *stores[SLOT_TREE_NODE_STORE_SIZE];
  1612. +};
  1613. +
  1614. +static struct kmem_cache *slot_tree_node_cachep;
  1615. +
  1616. +static struct sradix_tree_node *slot_tree_node_alloc(void)
  1617. +{
  1618. +   struct slot_tree_node *p;
  1619. +   p = kmem_cache_zalloc(slot_tree_node_cachep, GFP_KERNEL);
  1620. +   if (!p)
  1621. +       return NULL;
  1622. +
  1623. +   return &p->snode;
  1624. +}
  1625. +
  1626. +static void slot_tree_node_free(struct sradix_tree_node *node)
  1627. +{
  1628. +   struct slot_tree_node *p;
  1629. +
  1630. +   p = container_of(node, struct slot_tree_node, snode);
  1631. +   kmem_cache_free(slot_tree_node_cachep, p);
  1632. +}
  1633. +
  1634. +static void slot_tree_node_extend(struct sradix_tree_node *parent,
  1635. +                 struct sradix_tree_node *child)
  1636. +{
  1637. +   struct slot_tree_node *p, *c;
  1638. +
  1639. +   p = container_of(parent, struct slot_tree_node, snode);
  1640. +   c = container_of(child, struct slot_tree_node, snode);
  1641. +
  1642. +   p->size += c->size;
  1643. +}
  1644. +
  1645. +void slot_tree_node_assign(struct sradix_tree_node *node,
  1646. +              unsigned index, void *item)
  1647. +{
  1648. +   struct vma_slot *slot = item;
  1649. +   struct slot_tree_node *cur;
  1650. +
  1651. +   slot->snode = node;
  1652. +   slot->sindex = index;
  1653. +
  1654. +   while (node) {
  1655. +       cur = container_of(node, struct slot_tree_node, snode);
  1656. +       cur->size += slot->pages;
  1657. +       node = node->parent;
  1658. +   }
  1659. +}
  1660. +
  1661. +void slot_tree_node_rm(struct sradix_tree_node *node, unsigned offset)
  1662. +{
  1663. +   struct vma_slot *slot;
  1664. +   struct slot_tree_node *cur;
  1665. +   unsigned long pages;
  1666. +
  1667. +   if (node->height == 1) {
  1668. +       slot = node->stores[offset];
  1669. +       pages = slot->pages;
  1670. +   } else {
  1671. +       cur = container_of(node->stores[offset],
  1672. +                  struct slot_tree_node, snode);
  1673. +       pages = cur->size;
  1674. +   }
  1675. +
  1676. +   while (node) {
  1677. +       cur = container_of(node, struct slot_tree_node, snode);
  1678. +       cur->size -= pages;
  1679. +       node = node->parent;
  1680. +   }
  1681. +}
  1682. +
  1683. +unsigned long slot_iter_index;
  1684. +int slot_iter(void *item,  unsigned long height)
  1685. +{
  1686. +   struct slot_tree_node *node;
  1687. +   struct vma_slot *slot;
  1688. +
  1689. +   if (height == 1) {
  1690. +       slot = item;
  1691. +       if (slot_iter_index < slot->pages) {
  1692. +           /*in this one*/
  1693. +           return 1;
  1694. +       } else {
  1695. +           slot_iter_index -= slot->pages;
  1696. +           return 0;
  1697. +       }
  1698. +
  1699. +   } else {
  1700. +       node = container_of(item, struct slot_tree_node, snode);
  1701. +       if (slot_iter_index < node->size) {
  1702. +           /*in this one*/
  1703. +           return 1;
  1704. +       } else {
  1705. +           slot_iter_index -= node->size;
  1706. +           return 0;
  1707. +       }
  1708. +   }
  1709. +}
  1710. +
  1711. +
  1712. +static inline void slot_tree_init_root(struct sradix_tree_root *root)
  1713. +{
  1714. +   init_sradix_tree_root(root, SLOT_TREE_NODE_SHIFT);
  1715. +   root->alloc = slot_tree_node_alloc;
  1716. +   root->free = slot_tree_node_free;
  1717. +   root->extend = slot_tree_node_extend;
  1718. +   root->assign = slot_tree_node_assign;
  1719. +   root->rm = slot_tree_node_rm;
  1720. +}
  1721. +
  1722. +void slot_tree_init(void)
  1723. +{
  1724. +   slot_tree_node_cachep = kmem_cache_create("slot_tree_node",
  1725. +               sizeof(struct slot_tree_node), 0,
  1726. +               SLAB_PANIC | SLAB_RECLAIM_ACCOUNT,
  1727. +               NULL);
  1728. +}
  1729. +
  1730. +
  1731. +/* Each rung of this ladder is a list of VMAs having a same scan ratio */
  1732. +struct scan_rung {
  1733. +   //struct list_head scanned_list;
  1734. +   struct sradix_tree_root vma_root;
  1735. +   struct sradix_tree_root vma_root2;
  1736. +
  1737. +   struct vma_slot *current_scan;
  1738. +   unsigned long current_offset;
  1739. +
  1740. +   /*
  1741. +    * The initial value for current_offset, it should loop over
  1742. +    * [0~ step - 1] to let all slot have its chance to be scanned.
  1743. +    */
  1744. +   unsigned long offset_init;
  1745. +   unsigned long step; /* dynamic step for current_offset */
  1746. +   unsigned int flags;
  1747. +   unsigned long pages_to_scan;
  1748. +   //unsigned long fully_scanned_slots;
  1749. +   /*
  1750. +    * a little bit tricky - if cpu_time_ratio > 0, then the value is the
  1751. +    * the cpu time ratio it can spend in rung_i for every scan
  1752. +    * period. if < 0, then it is the cpu time ratio relative to the
  1753. +    * max cpu percentage user specified. Both in unit of
  1754. +    * 1/TIME_RATIO_SCALE
  1755. +    */
  1756. +   int cpu_ratio;
  1757. +
  1758. +   /*
  1759. +    * How long it will take for all slots in this rung to be fully
  1760. +    * scanned? If it's zero, we don't care about the cover time:
  1761. +    * it's fully scanned.
  1762. +    */
  1763. +   unsigned int cover_msecs;
  1764. +   //unsigned long vma_num;
  1765. +   //unsigned long pages; /* Sum of all slot's pages in rung */
  1766. +};
  1767. +
  1768. +/**
  1769. + * node of either the stable or unstale rbtree
  1770. + *
  1771. + */
  1772. +struct tree_node {
  1773. +   struct rb_node node; /* link in the main (un)stable rbtree */
  1774. +   struct rb_root sub_root; /* rb_root for sublevel collision rbtree */
  1775. +   u32 hash;
  1776. +   unsigned long count; /* TODO: merged with sub_root */
  1777. +   struct list_head all_list; /* all tree nodes in stable/unstable tree */
  1778. +};
  1779. +
  1780. +/**
  1781. + * struct stable_node - node of the stable rbtree
  1782. + * @node: rb node of this ksm page in the stable tree
  1783. + * @hlist: hlist head of rmap_items using this ksm page
  1784. + * @kpfn: page frame number of this ksm page
  1785. + */
  1786. +struct stable_node {
  1787. +   struct rb_node node; /* link in sub-rbtree */
  1788. +   struct tree_node *tree_node; /* it's tree node root in stable tree, NULL if it's in hell list */
  1789. +   struct hlist_head hlist;
  1790. +   unsigned long kpfn;
  1791. +   u32 hash_max; /* if ==0 then it's not been calculated yet */
  1792. +   struct list_head all_list; /* in a list for all stable nodes */
  1793. +};
  1794. +
  1795. +/**
  1796. + * struct node_vma - group rmap_items linked in a same stable
  1797. + * node together.
  1798. + */
  1799. +struct node_vma {
  1800. +   union {
  1801. +       struct vma_slot *slot;
  1802. +       unsigned long key;  /* slot is used as key sorted on hlist */
  1803. +   };
  1804. +   struct hlist_node hlist;
  1805. +   struct hlist_head rmap_hlist;
  1806. +   struct stable_node *head;
  1807. +};
  1808. +
  1809. +/**
  1810. + * struct rmap_item - reverse mapping item for virtual addresses
  1811. + * @rmap_list: next rmap_item in mm_slot's singly-linked rmap_list
  1812. + * @anon_vma: pointer to anon_vma for this mm,address, when in stable tree
  1813. + * @mm: the memory structure this rmap_item is pointing into
  1814. + * @address: the virtual address this rmap_item tracks (+ flags in low bits)
  1815. + * @node: rb node of this rmap_item in the unstable tree
  1816. + * @head: pointer to stable_node heading this list in the stable tree
  1817. + * @hlist: link into hlist of rmap_items hanging off that stable_node
  1818. + */
  1819. +struct rmap_item {
  1820. +   struct vma_slot *slot;
  1821. +   struct page *page;
  1822. +   unsigned long address;  /* + low bits used for flags below */
  1823. +   unsigned long hash_round;
  1824. +   unsigned long entry_index;
  1825. +   union {
  1826. +       struct {/* when in unstable tree */
  1827. +           struct rb_node node;
  1828. +           struct tree_node *tree_node;
  1829. +           u32 hash_max;
  1830. +       };
  1831. +       struct { /* when in stable tree */
  1832. +           struct node_vma *head;
  1833. +           struct hlist_node hlist;
  1834. +           struct anon_vma *anon_vma;
  1835. +       };
  1836. +   };
  1837. +} __attribute__((aligned(4)));
  1838. +
  1839. +struct rmap_list_entry {
  1840. +   union {
  1841. +       struct rmap_item *item;
  1842. +       unsigned long addr;
  1843. +   };
  1844. +   /* lowest bit is used for is_addr tag */
  1845. +} __attribute__((aligned(4))); /* 4 aligned to fit in to pages*/
  1846. +
  1847. +
  1848. +/* Basic data structure definition ends */
  1849. +
  1850. +
  1851. +/*
  1852. + * Flags for rmap_item to judge if it's listed in the stable/unstable tree.
  1853. + * The flags use the low bits of rmap_item.address
  1854. + */
  1855. +#define UNSTABLE_FLAG  0x1
  1856. +#define STABLE_FLAG    0x2
  1857. +#define get_rmap_addr(x)   ((x)->address & PAGE_MASK)
  1858. +
  1859. +/*
  1860. + * rmap_list_entry helpers
  1861. + */
  1862. +#define IS_ADDR_FLAG   1
  1863. +#define is_addr(ptr)       ((unsigned long)(ptr) & IS_ADDR_FLAG)
  1864. +#define set_is_addr(ptr)   ((ptr) |= IS_ADDR_FLAG)
  1865. +#define get_clean_addr(ptr)    (((ptr) & ~(__typeof__(ptr))IS_ADDR_FLAG))
  1866. +
  1867. +
  1868. +/*
  1869. + * High speed caches for frequently allocated and freed structs
  1870. + */
  1871. +static struct kmem_cache *rmap_item_cache;
  1872. +static struct kmem_cache *stable_node_cache;
  1873. +static struct kmem_cache *node_vma_cache;
  1874. +static struct kmem_cache *vma_slot_cache;
  1875. +static struct kmem_cache *tree_node_cache;
  1876. +#define UKSM_KMEM_CACHE(__struct, __flags) kmem_cache_create("uksm_"#__struct,\
  1877. +       sizeof(struct __struct), __alignof__(struct __struct),\
  1878. +       (__flags), NULL)
  1879. +
  1880. +/* Array of all scan_rung, uksm_scan_ladder[0] having the minimum scan ratio */
  1881. +#define SCAN_LADDER_SIZE 4
  1882. +static struct scan_rung uksm_scan_ladder[SCAN_LADDER_SIZE];
  1883. +
  1884. +/* The evaluation rounds uksmd has finished */
  1885. +static unsigned long long uksm_eval_round = 1;
  1886. +
  1887. +/*
  1888. + * we add 1 to this var when we consider we should rebuild the whole
  1889. + * unstable tree.
  1890. + */
  1891. +static unsigned long uksm_hash_round = 1;
  1892. +
  1893. +/*
  1894. + * How many times the whole memory is scanned.
  1895. + */
  1896. +static unsigned long long fully_scanned_round = 1;
  1897. +
  1898. +/* The total number of virtual pages of all vma slots */
  1899. +static u64 uksm_pages_total;
  1900. +
  1901. +/* The number of pages has been scanned since the start up */
  1902. +static u64 uksm_pages_scanned;
  1903. +
  1904. +static u64 scanned_virtual_pages;
  1905. +
  1906. +/* The number of pages has been scanned since last encode_benefit call */
  1907. +static u64 uksm_pages_scanned_last;
  1908. +
  1909. +/* If the scanned number is tooo large, we encode it here */
  1910. +static u64 pages_scanned_stored;
  1911. +
  1912. +static unsigned long pages_scanned_base;
  1913. +
  1914. +/* The number of nodes in the stable tree */
  1915. +static unsigned long uksm_pages_shared;
  1916. +
  1917. +/* The number of page slots additionally sharing those nodes */
  1918. +static unsigned long uksm_pages_sharing;
  1919. +
  1920. +/* The number of nodes in the unstable tree */
  1921. +static unsigned long uksm_pages_unshared;
  1922. +
  1923. +/*
  1924. + * Milliseconds ksmd should sleep between scans,
  1925. + * >= 100ms to be consistent with
  1926. + * scan_time_to_sleep_msec()
  1927. + */
  1928. +static unsigned int uksm_sleep_jiffies;
  1929. +
  1930. +/* The real value for the uksmd next sleep */
  1931. +static unsigned int uksm_sleep_real;
  1932. +
  1933. +/* Saved value for user input uksm_sleep_jiffies when it's enlarged */
  1934. +static unsigned int uksm_sleep_saved;
  1935. +
  1936. +/* Max percentage of cpu utilization ksmd can take to scan in one batch */
  1937. +static unsigned int uksm_max_cpu_percentage;
  1938. +
  1939. +static int uksm_cpu_governor;
  1940. +
  1941. +static char *uksm_cpu_governor_str[4] = { "full", "medium", "low", "quiet" };
  1942. +
  1943. +struct uksm_cpu_preset_s {
  1944. +   int cpu_ratio[SCAN_LADDER_SIZE];
  1945. +   unsigned int cover_msecs[SCAN_LADDER_SIZE];
  1946. +   unsigned int max_cpu; /* percentage */
  1947. +};
  1948. +
  1949. +struct uksm_cpu_preset_s uksm_cpu_preset[4] = {
  1950. +   { {20, 40, -2500, -10000}, {1000, 500, 200, 50}, 95},
  1951. +   { {20, 30, -2500, -10000}, {1000, 500, 400, 100}, 50},
  1952. +   { {10, 20, -5000, -10000}, {1500, 1000, 1000, 250}, 20},
  1953. +   { {10, 20, 40, 75}, {2000, 1000, 1000, 1000}, 1},
  1954. +};
  1955. +
  1956. +/* The default value for uksm_ema_page_time if it's not initialized */
  1957. +#define UKSM_PAGE_TIME_DEFAULT 500
  1958. +
  1959. +/*cost to scan one page by expotional moving average in nsecs */
  1960. +static unsigned long uksm_ema_page_time = UKSM_PAGE_TIME_DEFAULT;
  1961. +
  1962. +/* The expotional moving average alpha weight, in percentage. */
  1963. +#define EMA_ALPHA  20
  1964. +
  1965. +/*
  1966. + * The threshold used to filter out thrashing areas,
  1967. + * If it == 0, filtering is disabled, otherwise it's the percentage up-bound
  1968. + * of the thrashing ratio of all areas. Any area with a bigger thrashing ratio
  1969. + * will be considered as having a zero duplication ratio.
  1970. + */
  1971. +static unsigned int uksm_thrash_threshold = 50;
  1972. +
  1973. +/* How much dedup ratio is considered to be abundant*/
  1974. +static unsigned int uksm_abundant_threshold = 10;
  1975. +
  1976. +/* All slots having merged pages in this eval round. */
  1977. +struct list_head vma_slot_dedup = LIST_HEAD_INIT(vma_slot_dedup);
  1978. +
  1979. +/* How many times the ksmd has slept since startup */
  1980. +static unsigned long long uksm_sleep_times;
  1981. +
  1982. +#define UKSM_RUN_STOP  0
  1983. +#define UKSM_RUN_MERGE 1
  1984. +static unsigned int uksm_run = 1;
  1985. +
  1986. +static DECLARE_WAIT_QUEUE_HEAD(uksm_thread_wait);
  1987. +static DEFINE_MUTEX(uksm_thread_mutex);
  1988. +
  1989. +/*
  1990. + * List vma_slot_new is for newly created vma_slot waiting to be added by
  1991. + * ksmd. If one cannot be added(e.g. due to it's too small), it's moved to
  1992. + * vma_slot_noadd. vma_slot_del is the list for vma_slot whose corresponding
  1993. + * VMA has been removed/freed.
  1994. + */
  1995. +struct list_head vma_slot_new = LIST_HEAD_INIT(vma_slot_new);
  1996. +struct list_head vma_slot_noadd = LIST_HEAD_INIT(vma_slot_noadd);
  1997. +struct list_head vma_slot_del = LIST_HEAD_INIT(vma_slot_del);
  1998. +static DEFINE_SPINLOCK(vma_slot_list_lock);
  1999. +
  2000. +/* The unstable tree heads */
  2001. +static struct rb_root root_unstable_tree = RB_ROOT;
  2002. +
  2003. +/*
  2004. + * All tree_nodes are in a list to be freed at once when unstable tree is
  2005. + * freed after each scan round.
  2006. + */
  2007. +static struct list_head unstable_tree_node_list =
  2008. +               LIST_HEAD_INIT(unstable_tree_node_list);
  2009. +
  2010. +/* List contains all stable nodes */
  2011. +static struct list_head stable_node_list = LIST_HEAD_INIT(stable_node_list);
  2012. +
  2013. +/*
  2014. + * When the hash strength is changed, the stable tree must be delta_hashed and
  2015. + * re-structured. We use two set of below structs to speed up the
  2016. + * re-structuring of stable tree.
  2017. + */
  2018. +static struct list_head
  2019. +stable_tree_node_list[2] = {LIST_HEAD_INIT(stable_tree_node_list[0]),
  2020. +               LIST_HEAD_INIT(stable_tree_node_list[1])};
  2021. +
  2022. +static struct list_head *stable_tree_node_listp = &stable_tree_node_list[0];
  2023. +static struct rb_root root_stable_tree[2] = {RB_ROOT, RB_ROOT};
  2024. +static struct rb_root *root_stable_treep = &root_stable_tree[0];
  2025. +static unsigned long stable_tree_index;
  2026. +
  2027. +/* The hash strength needed to hash a full page */
  2028. +#define HASH_STRENGTH_FULL     (PAGE_SIZE / sizeof(u32))
  2029. +
  2030. +/* The hash strength needed for loop-back hashing */
  2031. +#define HASH_STRENGTH_MAX      (HASH_STRENGTH_FULL + 10)
  2032. +
  2033. +/* The random offsets in a page */
  2034. +static u32 *random_nums;
  2035. +
  2036. +/* The hash strength */
  2037. +static unsigned long hash_strength = HASH_STRENGTH_FULL >> 4;
  2038. +
  2039. +/* The delta value each time the hash strength increases or decreases */
  2040. +static unsigned long hash_strength_delta;
  2041. +#define HASH_STRENGTH_DELTA_MAX    5
  2042. +
  2043. +/* The time we have saved due to random_sample_hash */
  2044. +static u64 rshash_pos;
  2045. +
  2046. +/* The time we have wasted due to hash collision */
  2047. +static u64 rshash_neg;
  2048. +
  2049. +struct uksm_benefit {
  2050. +   u64 pos;
  2051. +   u64 neg;
  2052. +   u64 scanned;
  2053. +   unsigned long base;
  2054. +} benefit;
  2055. +
  2056. +/*
  2057. + * The relative cost of memcmp, compared to 1 time unit of random sample
  2058. + * hash, this value is tested when ksm module is initialized
  2059. + */
  2060. +static unsigned long memcmp_cost;
  2061. +
  2062. +static unsigned long  rshash_neg_cont_zero;
  2063. +static unsigned long  rshash_cont_obscure;
  2064. +
  2065. +/* The possible states of hash strength adjustment heuristic */
  2066. +enum rshash_states {
  2067. +       RSHASH_STILL,
  2068. +       RSHASH_TRYUP,
  2069. +       RSHASH_TRYDOWN,
  2070. +       RSHASH_NEW,
  2071. +       RSHASH_PRE_STILL,
  2072. +};
  2073. +
  2074. +/* The possible direction we are about to adjust hash strength */
  2075. +enum rshash_direct {
  2076. +   GO_UP,
  2077. +   GO_DOWN,
  2078. +   OBSCURE,
  2079. +   STILL,
  2080. +};
  2081. +
  2082. +/* random sampling hash state machine */
  2083. +static struct {
  2084. +   enum rshash_states state;
  2085. +   enum rshash_direct pre_direct;
  2086. +   u8 below_count;
  2087. +   /* Keep a lookup window of size 5, iff above_count/below_count > 3
  2088. +    * in this window we stop trying.
  2089. +    */
  2090. +   u8 lookup_window_index;
  2091. +   u64 stable_benefit;
  2092. +   unsigned long turn_point_down;
  2093. +   unsigned long turn_benefit_down;
  2094. +   unsigned long turn_point_up;
  2095. +   unsigned long turn_benefit_up;
  2096. +   unsigned long stable_point;
  2097. +} rshash_state;
  2098. +
  2099. +/*zero page hash table, hash_strength [0 ~ HASH_STRENGTH_MAX]*/
  2100. +static u32 *zero_hash_table;
  2101. +
  2102. +static inline struct node_vma *alloc_node_vma(void)
  2103. +{
  2104. +   struct node_vma *node_vma;
  2105. +   node_vma = kmem_cache_zalloc(node_vma_cache, GFP_KERNEL);
  2106. +   if (node_vma) {
  2107. +       INIT_HLIST_HEAD(&node_vma->rmap_hlist);
  2108. +       INIT_HLIST_NODE(&node_vma->hlist);
  2109. +   }
  2110. +   return node_vma;
  2111. +}
  2112. +
  2113. +static inline void free_node_vma(struct node_vma *node_vma)
  2114. +{
  2115. +   kmem_cache_free(node_vma_cache, node_vma);
  2116. +}
  2117. +
  2118. +
  2119. +static inline struct vma_slot *alloc_vma_slot(void)
  2120. +{
  2121. +   struct vma_slot *slot;
  2122. +
  2123. +   /*
  2124. +    * In case ksm is not initialized by now.
  2125. +    * Oops, we need to consider the call site of uksm_init() in the future.
  2126. +    */
  2127. +   if (!vma_slot_cache)
  2128. +       return NULL;
  2129. +
  2130. +   slot = kmem_cache_zalloc(vma_slot_cache, GFP_KERNEL);
  2131. +   if (slot) {
  2132. +       INIT_LIST_HEAD(&slot->slot_list);
  2133. +       INIT_LIST_HEAD(&slot->dedup_list);
  2134. +       slot->flags |= UKSM_SLOT_NEED_RERAND;
  2135. +   }
  2136. +   return slot;
  2137. +}
  2138. +
  2139. +static inline void free_vma_slot(struct vma_slot *vma_slot)
  2140. +{
  2141. +   kmem_cache_free(vma_slot_cache, vma_slot);
  2142. +}
  2143. +
  2144. +
  2145. +
  2146. +static inline struct rmap_item *alloc_rmap_item(void)
  2147. +{
  2148. +   struct rmap_item *rmap_item;
  2149. +
  2150. +   rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL);
  2151. +   if (rmap_item) {
  2152. +       /* bug on lowest bit is not clear for flag use */
  2153. +       BUG_ON(is_addr(rmap_item));
  2154. +   }
  2155. +   return rmap_item;
  2156. +}
  2157. +
  2158. +static inline void free_rmap_item(struct rmap_item *rmap_item)
  2159. +{
  2160. +   rmap_item->slot = NULL; /* debug safety */
  2161. +   kmem_cache_free(rmap_item_cache, rmap_item);
  2162. +}
  2163. +
  2164. +static inline struct stable_node *alloc_stable_node(void)
  2165. +{
  2166. +   struct stable_node *node;
  2167. +   node = kmem_cache_alloc(stable_node_cache, GFP_KERNEL | GFP_ATOMIC);
  2168. +   if (!node)
  2169. +       return NULL;
  2170. +
  2171. +   INIT_HLIST_HEAD(&node->hlist);
  2172. +   list_add(&node->all_list, &stable_node_list);
  2173. +   return node;
  2174. +}
  2175. +
  2176. +static inline void free_stable_node(struct stable_node *stable_node)
  2177. +{
  2178. +   list_del(&stable_node->all_list);
  2179. +   kmem_cache_free(stable_node_cache, stable_node);
  2180. +}
  2181. +
  2182. +static inline struct tree_node *alloc_tree_node(struct list_head *list)
  2183. +{
  2184. +   struct tree_node *node;
  2185. +   node = kmem_cache_zalloc(tree_node_cache, GFP_KERNEL | GFP_ATOMIC);
  2186. +   if (!node)
  2187. +       return NULL;
  2188. +
  2189. +   list_add(&node->all_list, list);
  2190. +   return node;
  2191. +}
  2192. +
  2193. +static inline void free_tree_node(struct tree_node *node)
  2194. +{
  2195. +   list_del(&node->all_list);
  2196. +   kmem_cache_free(tree_node_cache, node);
  2197. +}
  2198. +
  2199. +static void uksm_drop_anon_vma(struct rmap_item *rmap_item)
  2200. +{
  2201. +   struct anon_vma *anon_vma = rmap_item->anon_vma;
  2202. +
  2203. +   put_anon_vma(anon_vma);
  2204. +}
  2205. +
  2206. +
  2207. +/**
  2208. + * Remove a stable node from stable_tree, may unlink from its tree_node and
  2209. + * may remove its parent tree_node if no other stable node is pending.
  2210. + *
  2211. + * @stable_node    The node need to be removed
  2212. + * @unlink_rb      Will this node be unlinked from the rbtree?
  2213. + * @remove_tree_   node Will its tree_node be removed if empty?
  2214. + */
  2215. +static void remove_node_from_stable_tree(struct stable_node *stable_node,
  2216. +                    int unlink_rb,  int remove_tree_node)
  2217. +{
  2218. +   struct node_vma *node_vma;
  2219. +   struct rmap_item *rmap_item;
  2220. +   struct hlist_node *hlist, *rmap_hlist, *n;
  2221. +
  2222. +   if (!hlist_empty(&stable_node->hlist)) {
  2223. +       hlist_for_each_entry_safe(node_vma, hlist, n,
  2224. +                     &stable_node->hlist, hlist) {
  2225. +           hlist_for_each_entry(rmap_item, rmap_hlist,
  2226. +                        &node_vma->rmap_hlist, hlist) {
  2227. +               uksm_pages_sharing--;
  2228. +
  2229. +               uksm_drop_anon_vma(rmap_item);
  2230. +               rmap_item->address &= PAGE_MASK;
  2231. +           }
  2232. +           free_node_vma(node_vma);
  2233. +           cond_resched();
  2234. +       }
  2235. +
  2236. +       /* the last one is counted as shared */
  2237. +       uksm_pages_shared--;
  2238. +       uksm_pages_sharing++;
  2239. +   }
  2240. +
  2241. +   if (stable_node->tree_node && unlink_rb) {
  2242. +       rb_erase(&stable_node->node,
  2243. +            &stable_node->tree_node->sub_root);
  2244. +
  2245. +       if (RB_EMPTY_ROOT(&stable_node->tree_node->sub_root) &&
  2246. +           remove_tree_node) {
  2247. +           rb_erase(&stable_node->tree_node->node,
  2248. +                root_stable_treep);
  2249. +           free_tree_node(stable_node->tree_node);
  2250. +       } else {
  2251. +           stable_node->tree_node->count--;
  2252. +       }
  2253. +   }
  2254. +
  2255. +   free_stable_node(stable_node);
  2256. +}
  2257. +
  2258. +
  2259. +/*
  2260. + * get_uksm_page: checks if the page indicated by the stable node
  2261. + * is still its ksm page, despite having held no reference to it.
  2262. + * In which case we can trust the content of the page, and it
  2263. + * returns the gotten page; but if the page has now been zapped,
  2264. + * remove the stale node from the stable tree and return NULL.
  2265. + *
  2266. + * You would expect the stable_node to hold a reference to the ksm page.
  2267. + * But if it increments the page's count, swapping out has to wait for
  2268. + * ksmd to come around again before it can free the page, which may take
  2269. + * seconds or even minutes: much too unresponsive.  So instead we use a
  2270. + * "keyhole reference": access to the ksm page from the stable node peeps
  2271. + * out through its keyhole to see if that page still holds the right key,
  2272. + * pointing back to this stable node.  This relies on freeing a PageAnon
  2273. + * page to reset its page->mapping to NULL, and relies on no other use of
  2274. + * a page to put something that might look like our key in page->mapping.
  2275. + *
  2276. + * include/linux/pagemap.h page_cache_get_speculative() is a good reference,
  2277. + * but this is different - made simpler by uksm_thread_mutex being held, but
  2278. + * interesting for assuming that no other use of the struct page could ever
  2279. + * put our expected_mapping into page->mapping (or a field of the union which
  2280. + * coincides with page->mapping).  The RCU calls are not for KSM at all, but
  2281. + * to keep the page_count protocol described with page_cache_get_speculative.
  2282. + *
  2283. + * Note: it is possible that get_uksm_page() will return NULL one moment,
  2284. + * then page the next, if the page is in between page_freeze_refs() and
  2285. + * page_unfreeze_refs(): this shouldn't be a problem anywhere, the page
  2286. + * is on its way to being freed; but it is an anomaly to bear in mind.
  2287. + *
  2288. + * @unlink_rb:         if the removal of this node will firstly unlink from
  2289. + * its rbtree. stable_node_reinsert will prevent this when restructuring the
  2290. + * node from its old tree.
  2291. + *
  2292. + * @remove_tree_node:  if this is the last one of its tree_node, will the
  2293. + * tree_node be freed ? If we are inserting stable node, this tree_node may
  2294. + * be reused, so don't free it.
  2295. + */
  2296. +static struct page *get_uksm_page(struct stable_node *stable_node,
  2297. +                int unlink_rb, int remove_tree_node)
  2298. +{
  2299. +   struct page *page;
  2300. +   void *expected_mapping;
  2301. +
  2302. +   page = pfn_to_page(stable_node->kpfn);
  2303. +   expected_mapping = (void *)stable_node +
  2304. +               (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM);
  2305. +   rcu_read_lock();
  2306. +   if (page->mapping != expected_mapping)
  2307. +       goto stale;
  2308. +   if (!get_page_unless_zero(page))
  2309. +       goto stale;
  2310. +   if (page->mapping != expected_mapping) {
  2311. +       put_page(page);
  2312. +       goto stale;
  2313. +   }
  2314. +   rcu_read_unlock();
  2315. +   return page;
  2316. +stale:
  2317. +   rcu_read_unlock();
  2318. +   remove_node_from_stable_tree(stable_node, unlink_rb, remove_tree_node);
  2319. +
  2320. +   return NULL;
  2321. +}
  2322. +
  2323. +/*
  2324. + * Removing rmap_item from stable or unstable tree.
  2325. + * This function will clean the information from the stable/unstable tree.
  2326. + */
  2327. +static inline void remove_rmap_item_from_tree(struct rmap_item *rmap_item)
  2328. +{
  2329. +   if (rmap_item->address & STABLE_FLAG) {
  2330. +       struct stable_node *stable_node;
  2331. +       struct node_vma *node_vma;
  2332. +       struct page *page;
  2333. +
  2334. +       node_vma = rmap_item->head;
  2335. +       stable_node = node_vma->head;
  2336. +       page = get_uksm_page(stable_node, 1, 1);
  2337. +       if (!page)
  2338. +           goto out;
  2339. +
  2340. +       /*
  2341. +        * page lock is needed because it's racing with
  2342. +        * try_to_unmap_ksm(), etc.
  2343. +        */
  2344. +       lock_page(page);
  2345. +       hlist_del(&rmap_item->hlist);
  2346. +
  2347. +       if (hlist_empty(&node_vma->rmap_hlist)) {
  2348. +           hlist_del(&node_vma->hlist);
  2349. +           free_node_vma(node_vma);
  2350. +       }
  2351. +       unlock_page(page);
  2352. +
  2353. +       put_page(page);
  2354. +       if (hlist_empty(&stable_node->hlist)) {
  2355. +           /* do NOT call remove_node_from_stable_tree() here,
  2356. +            * it's possible for a forked rmap_item not in
  2357. +            * stable tree while the in-tree rmap_items were
  2358. +            * deleted.
  2359. +            */
  2360. +           uksm_pages_shared--;
  2361. +       } else
  2362. +           uksm_pages_sharing--;
  2363. +
  2364. +
  2365. +       uksm_drop_anon_vma(rmap_item);
  2366. +   } else if (rmap_item->address & UNSTABLE_FLAG) {
  2367. +       if (rmap_item->hash_round == uksm_hash_round) {
  2368. +
  2369. +           rb_erase(&rmap_item->node,
  2370. +                &rmap_item->tree_node->sub_root);
  2371. +           if (RB_EMPTY_ROOT(&rmap_item->tree_node->sub_root)) {
  2372. +               rb_erase(&rmap_item->tree_node->node,
  2373. +                    &root_unstable_tree);
  2374. +
  2375. +               free_tree_node(rmap_item->tree_node);
  2376. +           } else
  2377. +               rmap_item->tree_node->count--;
  2378. +       }
  2379. +       uksm_pages_unshared--;
  2380. +   }
  2381. +
  2382. +   rmap_item->address &= PAGE_MASK;
  2383. +   rmap_item->hash_max = 0;
  2384. +
  2385. +out:
  2386. +   cond_resched();     /* we're called from many long loops */
  2387. +}
  2388. +
  2389. +static inline int slot_in_uksm(struct vma_slot *slot)
  2390. +{
  2391. +   return list_empty(&slot->slot_list);
  2392. +}
  2393. +
  2394. +/*
  2395. + * Test if the mm is exiting
  2396. + */
  2397. +static inline bool uksm_test_exit(struct mm_struct *mm)
  2398. +{
  2399. +   return atomic_read(&mm->mm_users) == 0;
  2400. +}
  2401. +
  2402. +/**
  2403. + * Need to do two things:
  2404. + * 1. check if slot was moved to del list
  2405. + * 2. make sure the mmap_sem is manipulated under valid vma.
  2406. + *
  2407. + * My concern here is that in some cases, this may make
  2408. + * vma_slot_list_lock() waiters to serialized further by some
  2409. + * sem->wait_lock, can this really be expensive?
  2410. + *
  2411. + *
  2412. + * @return
  2413. + * 0: if successfully locked mmap_sem
  2414. + * -ENOENT: this slot was moved to del list
  2415. + * -EBUSY: vma lock failed
  2416. + */
  2417. +static int try_down_read_slot_mmap_sem(struct vma_slot *slot)
  2418. +{
  2419. +   struct vm_area_struct *vma;
  2420. +   struct mm_struct *mm;
  2421. +   struct rw_semaphore *sem;
  2422. +
  2423. +   spin_lock(&vma_slot_list_lock);
  2424. +
  2425. +   /* the slot_list was removed and inited from new list, when it enters
  2426. +    * uksm_list. If now it's not empty, then it must be moved to del list
  2427. +    */
  2428. +   if (!slot_in_uksm(slot)) {
  2429. +       spin_unlock(&vma_slot_list_lock);
  2430. +       return -ENOENT;
  2431. +   }
  2432. +
  2433. +   BUG_ON(slot->pages != vma_pages(slot->vma));
  2434. +   /* Ok, vma still valid */
  2435. +   vma = slot->vma;
  2436. +   mm = vma->vm_mm;
  2437. +   sem = &mm->mmap_sem;
  2438. +
  2439. +   if (uksm_test_exit(mm)) {
  2440. +       spin_unlock(&vma_slot_list_lock);
  2441. +       return -ENOENT;
  2442. +   }
  2443. +
  2444. +   if (down_read_trylock(sem)) {
  2445. +       spin_unlock(&vma_slot_list_lock);
  2446. +       return 0;
  2447. +   }
  2448. +
  2449. +   spin_unlock(&vma_slot_list_lock);
  2450. +   return -EBUSY;
  2451. +}
  2452. +
  2453. +static inline unsigned long
  2454. +vma_page_address(struct page *page, struct vm_area_struct *vma)
  2455. +{
  2456. +   pgoff_t pgoff = page->index << (PAGE_CACHE_SHIFT - PAGE_SHIFT);
  2457. +   unsigned long address;
  2458. +
  2459. +   address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
  2460. +   if (unlikely(address < vma->vm_start || address >= vma->vm_end)) {
  2461. +       /* page should be within @vma mapping range */
  2462. +       return -EFAULT;
  2463. +   }
  2464. +   return address;
  2465. +}
  2466. +
  2467. +
  2468. +/* return 0 on success with the item's mmap_sem locked */
  2469. +static inline int get_mergeable_page_lock_mmap(struct rmap_item *item)
  2470. +{
  2471. +   struct mm_struct *mm;
  2472. +   struct vma_slot *slot = item->slot;
  2473. +   int err = -EINVAL;
  2474. +
  2475. +   struct page *page;
  2476. +
  2477. +   /*
  2478. +    * try_down_read_slot_mmap_sem() returns non-zero if the slot
  2479. +    * has been removed by uksm_remove_vma().
  2480. +    */
  2481. +   if (try_down_read_slot_mmap_sem(slot))
  2482. +       return -EBUSY;
  2483. +
  2484. +   mm = slot->vma->vm_mm;
  2485. +
  2486. +   if (uksm_test_exit(mm))
  2487. +       goto failout_up;
  2488. +
  2489. +   page = item->page;
  2490. +   rcu_read_lock();
  2491. +   if (!get_page_unless_zero(page)) {
  2492. +       rcu_read_unlock();
  2493. +       goto failout_up;
  2494. +   }
  2495. +
  2496. +   /* No need to consider huge page here. */
  2497. +   if (item->slot->vma->anon_vma != page_anon_vma(page) ||
  2498. +       vma_page_address(page, item->slot->vma) != get_rmap_addr(item)) {
  2499. +       /*
  2500. +        * TODO:
  2501. +        * should we release this item becase of its stale page
  2502. +        * mapping?
  2503. +        */
  2504. +       put_page(page);
  2505. +       rcu_read_unlock();
  2506. +       goto failout_up;
  2507. +   }
  2508. +   rcu_read_unlock();
  2509. +   return 0;
  2510. +
  2511. +failout_up:
  2512. +   up_read(&mm->mmap_sem);
  2513. +   return err;
  2514. +}
  2515. +
  2516. +/*
  2517. + * What kind of VMA is considered ?
  2518. + */
  2519. +static inline int vma_can_enter(struct vm_area_struct *vma)
  2520. +{
  2521. +   return uksm_flags_can_scan(vma->vm_flags);
  2522. +}
  2523. +
  2524. +/*
  2525. + * Called whenever a fresh new vma is created A new vma_slot.
  2526. + * is created and inserted into a global list Must be called.
  2527. + * after vma is inserted to its mm                 .
  2528. + */
  2529. +void uksm_vma_add_new(struct vm_area_struct *vma)
  2530. +{
  2531. +   struct vma_slot *slot;
  2532. +
  2533. +   if (!vma_can_enter(vma)) {
  2534. +       vma->uksm_vma_slot = NULL;
  2535. +       return;
  2536. +   }
  2537. +
  2538. +   slot = alloc_vma_slot();
  2539. +   if (!slot) {
  2540. +       vma->uksm_vma_slot = NULL;
  2541. +       return;
  2542. +   }
  2543. +
  2544. +   vma->uksm_vma_slot = slot;
  2545. +   vma->vm_flags |= VM_MERGEABLE;
  2546. +   slot->vma = vma;
  2547. +   slot->mm = vma->vm_mm;
  2548. +   slot->ctime_j = jiffies;
  2549. +   slot->pages = vma_pages(vma);
  2550. +   spin_lock(&vma_slot_list_lock);
  2551. +   list_add_tail(&slot->slot_list, &vma_slot_new);
  2552. +   spin_unlock(&vma_slot_list_lock);
  2553. +}
  2554. +
  2555. +/*
  2556. + * Called after vma is unlinked from its mm
  2557. + */
  2558. +void uksm_remove_vma(struct vm_area_struct *vma)
  2559. +{
  2560. +   struct vma_slot *slot;
  2561. +
  2562. +   if (!vma->uksm_vma_slot)
  2563. +       return;
  2564. +
  2565. +   slot = vma->uksm_vma_slot;
  2566. +   spin_lock(&vma_slot_list_lock);
  2567. +   if (slot_in_uksm(slot)) {
  2568. +       /**
  2569. +        * This slot has been added by ksmd, so move to the del list
  2570. +        * waiting ksmd to free it.
  2571. +        */
  2572. +       list_add_tail(&slot->slot_list, &vma_slot_del);
  2573. +   } else {
  2574. +       /**
  2575. +        * It's still on new list. It's ok to free slot directly.
  2576. +        */
  2577. +       list_del(&slot->slot_list);
  2578. +       free_vma_slot(slot);
  2579. +   }
  2580. +   spin_unlock(&vma_slot_list_lock);
  2581. +   vma->uksm_vma_slot = NULL;
  2582. +}
  2583. +
  2584. +/*   32/3 < they < 32/2 */
  2585. +#define shiftl 8
  2586. +#define shiftr 12
  2587. +
  2588. +#define HASH_FROM_TO(from, to)                 \
  2589. +for (index = from; index < to; index++) {      \
  2590. +   pos = random_nums[index];           \
  2591. +   hash += key[pos];               \
  2592. +   hash += (hash << shiftl);           \
  2593. +   hash ^= (hash >> shiftr);           \
  2594. +}
  2595. +
  2596. +
  2597. +#define HASH_FROM_DOWN_TO(from, to)            \
  2598. +for (index = from - 1; index >= to; index--) {     \
  2599. +   hash ^= (hash >> shiftr);           \
  2600. +   hash ^= (hash >> (shiftr*2));           \
  2601. +   hash -= (hash << shiftl);           \
  2602. +   hash += (hash << (shiftl*2));           \
  2603. +   pos = random_nums[index];           \
  2604. +   hash -= key[pos];               \
  2605. +}
  2606. +
  2607. +/*
  2608. + * The main random sample hash function.
  2609. + */
  2610. +static u32 random_sample_hash(void *addr, u32 hash_strength)
  2611. +{
  2612. +   u32 hash = 0xdeadbeef;
  2613. +   int index, pos, loop = hash_strength;
  2614. +   u32 *key = (u32 *)addr;
  2615. +
  2616. +   if (loop > HASH_STRENGTH_FULL)
  2617. +       loop = HASH_STRENGTH_FULL;
  2618. +
  2619. +   HASH_FROM_TO(0, loop);
  2620. +
  2621. +   if (hash_strength > HASH_STRENGTH_FULL) {
  2622. +       loop = hash_strength - HASH_STRENGTH_FULL;
  2623. +       HASH_FROM_TO(0, loop);
  2624. +   }
  2625. +
  2626. +   return hash;
  2627. +}
  2628. +
  2629. +
  2630. +/**
  2631. + * It's used when hash strength is adjusted
  2632. + *
  2633. + * @addr The page's virtual address
  2634. + * @from The original hash strength
  2635. + * @to   The hash strength changed to
  2636. + * @hash The hash value generated with "from" hash value
  2637. + *
  2638. + * return the hash value
  2639. + */
  2640. +static u32 delta_hash(void *addr, int from, int to, u32 hash)
  2641. +{
  2642. +   u32 *key = (u32 *)addr;
  2643. +   int index, pos; /* make sure they are int type */
  2644. +
  2645. +   if (to > from) {
  2646. +       if (from >= HASH_STRENGTH_FULL) {
  2647. +           from -= HASH_STRENGTH_FULL;
  2648. +           to -= HASH_STRENGTH_FULL;
  2649. +           HASH_FROM_TO(from, to);
  2650. +       } else if (to <= HASH_STRENGTH_FULL) {
  2651. +           HASH_FROM_TO(from, to);
  2652. +       } else {
  2653. +           HASH_FROM_TO(from, HASH_STRENGTH_FULL);
  2654. +           HASH_FROM_TO(0, to - HASH_STRENGTH_FULL);
  2655. +       }
  2656. +   } else {
  2657. +       if (from <= HASH_STRENGTH_FULL) {
  2658. +           HASH_FROM_DOWN_TO(from, to);
  2659. +       } else if (to >= HASH_STRENGTH_FULL) {
  2660. +           from -= HASH_STRENGTH_FULL;
  2661. +           to -= HASH_STRENGTH_FULL;
  2662. +           HASH_FROM_DOWN_TO(from, to);
  2663. +       } else {
  2664. +           HASH_FROM_DOWN_TO(from - HASH_STRENGTH_FULL, 0);
  2665. +           HASH_FROM_DOWN_TO(HASH_STRENGTH_FULL, to);
  2666. +       }
  2667. +   }
  2668. +
  2669. +   return hash;
  2670. +}
  2671. +
  2672. +
  2673. +
  2674. +
  2675. +#define CAN_OVERFLOW_U64(x, delta) (U64_MAX - (x) < (delta))
  2676. +
  2677. +/**
  2678. + *
  2679. + * Called when: rshash_pos or rshash_neg is about to overflow or a scan round
  2680. + * has finished.
  2681. + *
  2682. + * return 0 if no page has been scanned since last call, 1 otherwise.
  2683. + */
  2684. +static inline int encode_benefit(void)
  2685. +{
  2686. +   u64 scanned_delta, pos_delta, neg_delta;
  2687. +   unsigned long base = benefit.base;
  2688. +
  2689. +   scanned_delta = uksm_pages_scanned - uksm_pages_scanned_last;
  2690. +
  2691. +   if (!scanned_delta)
  2692. +       return 0;
  2693. +
  2694. +   scanned_delta >>= base;
  2695. +   pos_delta = rshash_pos >> base;
  2696. +   neg_delta = rshash_neg >> base;
  2697. +
  2698. +   if (CAN_OVERFLOW_U64(benefit.pos, pos_delta) ||
  2699. +       CAN_OVERFLOW_U64(benefit.neg, neg_delta) ||
  2700. +       CAN_OVERFLOW_U64(benefit.scanned, scanned_delta)) {
  2701. +       benefit.scanned >>= 1;
  2702. +       benefit.neg >>= 1;
  2703. +       benefit.pos >>= 1;
  2704. +       benefit.base++;
  2705. +       scanned_delta >>= 1;
  2706. +       pos_delta >>= 1;
  2707. +       neg_delta >>= 1;
  2708. +   }
  2709. +
  2710. +   benefit.pos += pos_delta;
  2711. +   benefit.neg += neg_delta;
  2712. +   benefit.scanned += scanned_delta;
  2713. +
  2714. +   BUG_ON(!benefit.scanned);
  2715. +
  2716. +   rshash_pos = rshash_neg = 0;
  2717. +   uksm_pages_scanned_last = uksm_pages_scanned;
  2718. +
  2719. +   return 1;
  2720. +}
  2721. +
  2722. +static inline void reset_benefit(void)
  2723. +{
  2724. +   benefit.pos = 0;
  2725. +   benefit.neg = 0;
  2726. +   benefit.base = 0;
  2727. +   benefit.scanned = 0;
  2728. +}
  2729. +
  2730. +static inline void inc_rshash_pos(unsigned long delta)
  2731. +{
  2732. +   if (CAN_OVERFLOW_U64(rshash_pos, delta))
  2733. +       encode_benefit();
  2734. +
  2735. +   rshash_pos += delta;
  2736. +}
  2737. +
  2738. +static inline void inc_rshash_neg(unsigned long delta)
  2739. +{
  2740. +   if (CAN_OVERFLOW_U64(rshash_neg, delta))
  2741. +       encode_benefit();
  2742. +
  2743. +   rshash_neg += delta;
  2744. +}
  2745. +
  2746. +
  2747. +static inline u32 page_hash(struct page *page, unsigned long hash_strength,
  2748. +               int cost_accounting)
  2749. +{
  2750. +   u32 val;
  2751. +   unsigned long delta;
  2752. +
  2753. +   void *addr = kmap_atomic(page);
  2754. +
  2755. +   val = random_sample_hash(addr, hash_strength);
  2756. +   kunmap_atomic(addr);
  2757. +
  2758. +   if (cost_accounting) {
  2759. +       if (HASH_STRENGTH_FULL > hash_strength)
  2760. +           delta = HASH_STRENGTH_FULL - hash_strength;
  2761. +       else
  2762. +           delta = 0;
  2763. +
  2764. +       inc_rshash_pos(delta);
  2765. +   }
  2766. +
  2767. +   return val;
  2768. +}
  2769. +
  2770. +static int memcmp_pages(struct page *page1, struct page *page2,
  2771. +           int cost_accounting)
  2772. +{
  2773. +   char *addr1, *addr2;
  2774. +   int ret;
  2775. +
  2776. +   addr1 = kmap_atomic(page1);
  2777. +   addr2 = kmap_atomic(page2);
  2778. +   ret = memcmp(addr1, addr2, PAGE_SIZE);
  2779. +   kunmap_atomic(addr2);
  2780. +   kunmap_atomic(addr1);
  2781. +
  2782. +   if (cost_accounting)
  2783. +       inc_rshash_neg(memcmp_cost);
  2784. +
  2785. +   return ret;
  2786. +}
  2787. +
  2788. +static inline int pages_identical(struct page *page1, struct page *page2)
  2789. +{
  2790. +   return !memcmp_pages(page1, page2, 0);
  2791. +}
  2792. +
  2793. +static inline int is_page_full_zero(struct page *page)
  2794. +{
  2795. +   char *addr;
  2796. +   int ret;
  2797. +
  2798. +   addr = kmap_atomic(page);
  2799. +   ret = is_full_zero(addr, PAGE_SIZE);
  2800. +   kunmap_atomic(addr);
  2801. +
  2802. +   return ret;
  2803. +}
  2804. +
  2805. +static int write_protect_page(struct vm_area_struct *vma, struct page *page,
  2806. +                 pte_t *orig_pte, pte_t *old_pte)
  2807. +{
  2808. +   struct mm_struct *mm = vma->vm_mm;
  2809. +   unsigned long addr;
  2810. +   pte_t *ptep;
  2811. +   spinlock_t *ptl;
  2812. +   int swapped;
  2813. +   int err = -EFAULT;
  2814. +
  2815. +   addr = page_address_in_vma(page, vma);
  2816. +   if (addr == -EFAULT)
  2817. +       goto out;
  2818. +
  2819. +   BUG_ON(PageTransCompound(page));
  2820. +   ptep = page_check_address(page, mm, addr, &ptl, 0);
  2821. +   if (!ptep)
  2822. +       goto out;
  2823. +
  2824. +   if (old_pte)
  2825. +       *old_pte = *ptep;
  2826. +
  2827. +   if (pte_write(*ptep) || pte_dirty(*ptep)) {
  2828. +       pte_t entry;
  2829. +
  2830. +       swapped = PageSwapCache(page);
  2831. +       flush_cache_page(vma, addr, page_to_pfn(page));
  2832. +       /*
  2833. +        * Ok this is tricky, when get_user_pages_fast() run it doesnt
  2834. +        * take any lock, therefore the check that we are going to make
  2835. +        * with the pagecount against the mapcount is racey and
  2836. +        * O_DIRECT can happen right after the check.
  2837. +        * So we clear the pte and flush the tlb before the check
  2838. +        * this assure us that no O_DIRECT can happen after the check
  2839. +        * or in the middle of the check.
  2840. +        */
  2841. +       entry = ptep_clear_flush(vma, addr, ptep);
  2842. +       /*
  2843. +        * Check that no O_DIRECT or similar I/O is in progress on the
  2844. +        * page
  2845. +        */
  2846. +       if (page_mapcount(page) + 1 + swapped != page_count(page)) {
  2847. +           set_pte_at(mm, addr, ptep, entry);
  2848. +           goto out_unlock;
  2849. +       }
  2850. +       if (pte_dirty(entry))
  2851. +           set_page_dirty(page);
  2852. +       entry = pte_mkclean(pte_wrprotect(entry));
  2853. +       set_pte_at_notify(mm, addr, ptep, entry);
  2854. +   }
  2855. +   *orig_pte = *ptep;
  2856. +   err = 0;
  2857. +
  2858. +out_unlock:
  2859. +   pte_unmap_unlock(ptep, ptl);
  2860. +out:
  2861. +   return err;
  2862. +}
  2863. +
  2864. +#define MERGE_ERR_PGERR        1 /* the page is invalid cannot continue */
  2865. +#define MERGE_ERR_COLLI        2 /* there is a collision */
  2866. +#define MERGE_ERR_COLLI_MAX    3 /* collision at the max hash strength */
  2867. +#define MERGE_ERR_CHANGED  4 /* the page has changed since last hash */
  2868. +
  2869. +
  2870. +/**
  2871. + * replace_page - replace page in vma by new ksm page
  2872. + * @vma:      vma that holds the pte pointing to page
  2873. + * @page:     the page we are replacing by kpage
  2874. + * @kpage:    the ksm page we replace page by
  2875. + * @orig_pte: the original value of the pte
  2876. + *
  2877. + * Returns 0 on success, MERGE_ERR_PGERR on failure.
  2878. + */
  2879. +static int replace_page(struct vm_area_struct *vma, struct page *page,
  2880. +           struct page *kpage, pte_t orig_pte)
  2881. +{
  2882. +   struct mm_struct *mm = vma->vm_mm;
  2883. +   pgd_t *pgd;
  2884. +   pud_t *pud;
  2885. +   pmd_t *pmd;
  2886. +   pte_t *ptep;
  2887. +   spinlock_t *ptl;
  2888. +   pte_t entry;
  2889. +
  2890. +   unsigned long addr;
  2891. +   int err = MERGE_ERR_PGERR;
  2892. +
  2893. +   addr = page_address_in_vma(page, vma);
  2894. +   if (addr == -EFAULT)
  2895. +       goto out;
  2896. +
  2897. +   pgd = pgd_offset(mm, addr);
  2898. +   if (!pgd_present(*pgd))
  2899. +       goto out;
  2900. +
  2901. +   pud = pud_offset(pgd, addr);
  2902. +   if (!pud_present(*pud))
  2903. +       goto out;
  2904. +
  2905. +   pmd = pmd_offset(pud, addr);
  2906. +   BUG_ON(pmd_trans_huge(*pmd));
  2907. +   if (!pmd_present(*pmd))
  2908. +       goto out;
  2909. +
  2910. +   ptep = pte_offset_map_lock(mm, pmd, addr, &ptl);
  2911. +   if (!pte_same(*ptep, orig_pte)) {
  2912. +       pte_unmap_unlock(ptep, ptl);
  2913. +       goto out;
  2914. +   }
  2915. +
  2916. +   flush_cache_page(vma, addr, pte_pfn(*ptep));
  2917. +   ptep_clear_flush(vma, addr, ptep);
  2918. +   entry = mk_pte(kpage, vma->vm_page_prot);
  2919. +
  2920. +   /* special treatment is needed for zero_page */
  2921. +   if ((page_to_pfn(kpage) == uksm_zero_pfn) ||
  2922. +               (page_to_pfn(kpage) == zero_pfn))
  2923. +       entry = pte_mkspecial(entry);
  2924. +   else {
  2925. +       get_page(kpage);
  2926. +       page_add_anon_rmap(kpage, vma, addr);
  2927. +   }
  2928. +
  2929. +   set_pte_at_notify(mm, addr, ptep, entry);
  2930. +
  2931. +   page_remove_rmap(page);
  2932. +   if (!page_mapped(page))
  2933. +       try_to_free_swap(page);
  2934. +   put_page(page);
  2935. +
  2936. +   pte_unmap_unlock(ptep, ptl);
  2937. +   err = 0;
  2938. +out:
  2939. +   return err;
  2940. +}
  2941. +
  2942. +
  2943. +/**
  2944. + *  Fully hash a page with HASH_STRENGTH_MAX return a non-zero hash value. The
  2945. + *  zero hash value at HASH_STRENGTH_MAX is used to indicated that its
  2946. + *  hash_max member has not been calculated.
  2947. + *
  2948. + * @page The page needs to be hashed
  2949. + * @hash_old The hash value calculated with current hash strength
  2950. + *
  2951. + * return the new hash value calculated at HASH_STRENGTH_MAX
  2952. + */
  2953. +static inline u32 page_hash_max(struct page *page, u32 hash_old)
  2954. +{
  2955. +   u32 hash_max = 0;
  2956. +   void *addr;
  2957. +
  2958. +   addr = kmap_atomic(page);
  2959. +   hash_max = delta_hash(addr, hash_strength,
  2960. +                 HASH_STRENGTH_MAX, hash_old);
  2961. +
  2962. +   kunmap_atomic(addr);
  2963. +
  2964. +   if (!hash_max)
  2965. +       hash_max = 1;
  2966. +
  2967. +   inc_rshash_neg(HASH_STRENGTH_MAX - hash_strength);
  2968. +   return hash_max;
  2969. +}
  2970. +
  2971. +/*
  2972. + * We compare the hash again, to ensure that it is really a hash collision
  2973. + * instead of being caused by page write.
  2974. + */
  2975. +static inline int check_collision(struct rmap_item *rmap_item,
  2976. +                 u32 hash)
  2977. +{
  2978. +   int err;
  2979. +   struct page *page = rmap_item->page;
  2980. +
  2981. +   /* if this rmap_item has already been hash_maxed, then the collision
  2982. +    * must appears in the second-level rbtree search. In this case we check
  2983. +    * if its hash_max value has been changed. Otherwise, the collision
  2984. +    * happens in the first-level rbtree search, so we check against it's
  2985. +    * current hash value.
  2986. +    */
  2987. +   if (rmap_item->hash_max) {
  2988. +       inc_rshash_neg(memcmp_cost);
  2989. +       inc_rshash_neg(HASH_STRENGTH_MAX - hash_strength);
  2990. +
  2991. +       if (rmap_item->hash_max == page_hash_max(page, hash))
  2992. +           err = MERGE_ERR_COLLI;
  2993. +       else
  2994. +           err = MERGE_ERR_CHANGED;
  2995. +   } else {
  2996. +       inc_rshash_neg(memcmp_cost + hash_strength);
  2997. +
  2998. +       if (page_hash(page, hash_strength, 0) == hash)
  2999. +           err = MERGE_ERR_COLLI;
  3000. +       else
  3001. +           err = MERGE_ERR_CHANGED;
  3002. +   }
  3003. +
  3004. +   return err;
  3005. +}
  3006. +
  3007. +static struct page *page_trans_compound_anon(struct page *page)
  3008. +{
  3009. +   if (PageTransCompound(page)) {
  3010. +       struct page *head = compound_trans_head(page);
  3011. +       /*
  3012. +        * head may actually be splitted and freed from under
  3013. +        * us but it's ok here.
  3014. +        */
  3015. +       if (PageAnon(head))
  3016. +           return head;
  3017. +   }
  3018. +   return NULL;
  3019. +}
  3020. +
  3021. +static int page_trans_compound_anon_split(struct page *page)
  3022. +{
  3023. +   int ret = 0;
  3024. +   struct page *transhuge_head = page_trans_compound_anon(page);
  3025. +   if (transhuge_head) {
  3026. +       /* Get the reference on the head to split it. */
  3027. +       if (get_page_unless_zero(transhuge_head)) {
  3028. +           /*
  3029. +            * Recheck we got the reference while the head
  3030. +            * was still anonymous.
  3031. +            */
  3032. +           if (PageAnon(transhuge_head))
  3033. +               ret = split_huge_page(transhuge_head);
  3034. +           else
  3035. +               /*
  3036. +                * Retry later if split_huge_page run
  3037. +                * from under us.
  3038. +                */
  3039. +               ret = 1;
  3040. +           put_page(transhuge_head);
  3041. +       } else
  3042. +           /* Retry later if split_huge_page run from under us. */
  3043. +           ret = 1;
  3044. +   }
  3045. +   return ret;
  3046. +}
  3047. +
  3048. +/**
  3049. + * Try to merge a rmap_item.page with a kpage in stable node. kpage must
  3050. + * already be a ksm page.
  3051. + *
  3052. + * @return 0 if the pages were merged, -EFAULT otherwise.
  3053. + */
  3054. +static int try_to_merge_with_uksm_page(struct rmap_item *rmap_item,
  3055. +                     struct page *kpage, u32 hash)
  3056. +{
  3057. +   struct vm_area_struct *vma = rmap_item->slot->vma;
  3058. +   struct mm_struct *mm = vma->vm_mm;
  3059. +   pte_t orig_pte = __pte(0);
  3060. +   int err = MERGE_ERR_PGERR;
  3061. +   struct page *page;
  3062. +
  3063. +   if (uksm_test_exit(mm))
  3064. +       goto out;
  3065. +
  3066. +   page = rmap_item->page;
  3067. +
  3068. +   if (page == kpage) { /* ksm page forked */
  3069. +       err = 0;
  3070. +       goto out;
  3071. +   }
  3072. +
  3073. +   if (PageTransCompound(page) && page_trans_compound_anon_split(page))
  3074. +       goto out;
  3075. +   BUG_ON(PageTransCompound(page));
  3076. +
  3077. +   if (!PageAnon(page) || !PageKsm(kpage))
  3078. +       goto out;
  3079. +
  3080. +   /*
  3081. +    * We need the page lock to read a stable PageSwapCache in
  3082. +    * write_protect_page().  We use trylock_page() instead of
  3083. +    * lock_page() because we don't want to wait here - we
  3084. +    * prefer to continue scanning and merging different pages,
  3085. +    * then come back to this page when it is unlocked.
  3086. +    */
  3087. +   if (!trylock_page(page))
  3088. +       goto out;
  3089. +   /*
  3090. +    * If this anonymous page is mapped only here, its pte may need
  3091. +    * to be write-protected.  If it's mapped elsewhere, all of its
  3092. +    * ptes are necessarily already write-protected.  But in either
  3093. +    * case, we need to lock and check page_count is not raised.
  3094. +    */
  3095. +   if (write_protect_page(vma, page, &orig_pte, NULL) == 0) {
  3096. +       if (pages_identical(page, kpage))
  3097. +           err = replace_page(vma, page, kpage, orig_pte);
  3098. +       else
  3099. +           err = check_collision(rmap_item, hash);
  3100. +   }
  3101. +
  3102. +   if ((vma->vm_flags & VM_LOCKED) && kpage && !err) {
  3103. +       munlock_vma_page(page);
  3104. +       if (!PageMlocked(kpage)) {
  3105. +           unlock_page(page);
  3106. +           lock_page(kpage);
  3107. +           mlock_vma_page(kpage);
  3108. +           page = kpage;       /* for final unlock */
  3109. +       }
  3110. +   }
  3111. +
  3112. +   unlock_page(page);
  3113. +out:
  3114. +   return err;
  3115. +}
  3116. +
  3117. +
  3118. +
  3119. +/**
  3120. + * If two pages fail to merge in try_to_merge_two_pages, then we have a chance
  3121. + * to restore a page mapping that has been changed in try_to_merge_two_pages.
  3122. + *
  3123. + * @return 0 on success.
  3124. + */
  3125. +static int restore_uksm_page_pte(struct vm_area_struct *vma, unsigned long addr,
  3126. +                pte_t orig_pte, pte_t wprt_pte)
  3127. +{
  3128. +   struct mm_struct *mm = vma->vm_mm;
  3129. +   pgd_t *pgd;
  3130. +   pud_t *pud;
  3131. +   pmd_t *pmd;
  3132. +   pte_t *ptep;
  3133. +   spinlock_t *ptl;
  3134. +
  3135. +   int err = -EFAULT;
  3136. +
  3137. +   pgd = pgd_offset(mm, addr);
  3138. +   if (!pgd_present(*pgd))
  3139. +       goto out;
  3140. +
  3141. +   pud = pud_offset(pgd, addr);
  3142. +   if (!pud_present(*pud))
  3143. +       goto out;
  3144. +
  3145. +   pmd = pmd_offset(pud, addr);
  3146. +   if (!pmd_present(*pmd))
  3147. +       goto out;
  3148. +
  3149. +   ptep = pte_offset_map_lock(mm, pmd, addr, &ptl);
  3150. +   if (!pte_same(*ptep, wprt_pte)) {
  3151. +       /* already copied, let it be */
  3152. +       pte_unmap_unlock(ptep, ptl);
  3153. +       goto out;
  3154. +   }
  3155. +
  3156. +   /*
  3157. +    * Good boy, still here. When we still get the ksm page, it does not
  3158. +    * return to the free page pool, there is no way that a pte was changed
  3159. +    * to other page and gets back to this page. And remind that ksm page
  3160. +    * do not reuse in do_wp_page(). So it's safe to restore the original
  3161. +    * pte.
  3162. +    */
  3163. +   flush_cache_page(vma, addr, pte_pfn(*ptep));
  3164. +   ptep_clear_flush(vma, addr, ptep);
  3165. +   set_pte_at_notify(mm, addr, ptep, orig_pte);
  3166. +
  3167. +   pte_unmap_unlock(ptep, ptl);
  3168. +   err = 0;
  3169. +out:
  3170. +   return err;
  3171. +}
  3172. +
  3173. +/**
  3174. + * try_to_merge_two_pages() - take two identical pages and prepare
  3175. + * them to be merged into one page(rmap_item->page)
  3176. + *
  3177. + * @return 0 if we successfully merged two identical pages into
  3178. + *         one ksm page. MERGE_ERR_COLLI if it's only a hash collision
  3179. + *         search in rbtree. MERGE_ERR_CHANGED if rmap_item has been
  3180. + *         changed since it's hashed. MERGE_ERR_PGERR otherwise.
  3181. + *
  3182. + */
  3183. +static int try_to_merge_two_pages(struct rmap_item *rmap_item,
  3184. +                 struct rmap_item *tree_rmap_item,
  3185. +                 u32 hash)
  3186. +{
  3187. +   pte_t orig_pte1 = __pte(0), orig_pte2 = __pte(0);
  3188. +   pte_t wprt_pte1 = __pte(0), wprt_pte2 = __pte(0);
  3189. +   struct vm_area_struct *vma1 = rmap_item->slot->vma;
  3190. +   struct vm_area_struct *vma2 = tree_rmap_item->slot->vma;
  3191. +   struct page *page = rmap_item->page;
  3192. +   struct page *tree_page = tree_rmap_item->page;
  3193. +   int err = MERGE_ERR_PGERR;
  3194. +   struct address_space *saved_mapping;
  3195. +
  3196. +
  3197. +   if (rmap_item->page == tree_rmap_item->page)
  3198. +       goto out;
  3199. +
  3200. +   if (PageTransCompound(page) && page_trans_compound_anon_split(page))
  3201. +       goto out;
  3202. +   BUG_ON(PageTransCompound(page));
  3203. +
  3204. +   if (PageTransCompound(tree_page) && page_trans_compound_anon_split(tree_page))
  3205. +       goto out;
  3206. +   BUG_ON(PageTransCompound(tree_page));
  3207. +
  3208. +   if (!PageAnon(page) || !PageAnon(tree_page))
  3209. +       goto out;
  3210. +
  3211. +   if (!trylock_page(page))
  3212. +       goto out;
  3213. +
  3214. +
  3215. +   if (write_protect_page(vma1, page, &wprt_pte1, &orig_pte1) != 0) {
  3216. +       unlock_page(page);
  3217. +       goto out;
  3218. +   }
  3219. +
  3220. +   /*
  3221. +    * While we hold page lock, upgrade page from
  3222. +    * PageAnon+anon_vma to PageKsm+NULL stable_node:
  3223. +    * stable_tree_insert() will update stable_node.
  3224. +    */
  3225. +   saved_mapping = page->mapping;
  3226. +   set_page_stable_node(page, NULL);
  3227. +   mark_page_accessed(page);
  3228. +   unlock_page(page);
  3229. +
  3230. +   if (!trylock_page(tree_page))
  3231. +       goto restore_out;
  3232. +
  3233. +   if (write_protect_page(vma2, tree_page, &wprt_pte2, &orig_pte2) != 0) {
  3234. +       unlock_page(tree_page);
  3235. +       goto restore_out;
  3236. +   }
  3237. +
  3238. +   if (pages_identical(page, tree_page)) {
  3239. +       err = replace_page(vma2, tree_page, page, wprt_pte2);
  3240. +       if (err) {
  3241. +           unlock_page(tree_page);
  3242. +           goto restore_out;
  3243. +       }
  3244. +
  3245. +       if ((vma2->vm_flags & VM_LOCKED)) {
  3246. +           munlock_vma_page(tree_page);
  3247. +           if (!PageMlocked(page)) {
  3248. +               unlock_page(tree_page);
  3249. +               lock_page(page);
  3250. +               mlock_vma_page(page);
  3251. +               tree_page = page; /* for final unlock */
  3252. +           }
  3253. +       }
  3254. +
  3255. +       unlock_page(tree_page);
  3256. +
  3257. +       goto out; /* success */
  3258. +
  3259. +   } else {
  3260. +       if (tree_rmap_item->hash_max &&
  3261. +           tree_rmap_item->hash_max == rmap_item->hash_max) {
  3262. +           err = MERGE_ERR_COLLI_MAX;
  3263. +       } else if (page_hash(page, hash_strength, 0) ==
  3264. +           page_hash(tree_page, hash_strength, 0)) {
  3265. +           inc_rshash_neg(memcmp_cost + hash_strength * 2);
  3266. +           err = MERGE_ERR_COLLI;
  3267. +       } else {
  3268. +           err = MERGE_ERR_CHANGED;
  3269. +       }
  3270. +
  3271. +       unlock_page(tree_page);
  3272. +   }
  3273. +
  3274. +restore_out:
  3275. +   lock_page(page);
  3276. +   if (!restore_uksm_page_pte(vma1, get_rmap_addr(rmap_item),
  3277. +                 orig_pte1, wprt_pte1))
  3278. +       page->mapping = saved_mapping;
  3279. +
  3280. +   unlock_page(page);
  3281. +out:
  3282. +   return err;
  3283. +}
  3284. +
  3285. +static inline int hash_cmp(u32 new_val, u32 node_val)
  3286. +{
  3287. +   if (new_val > node_val)
  3288. +       return 1;
  3289. +   else if (new_val < node_val)
  3290. +       return -1;
  3291. +   else
  3292. +       return 0;
  3293. +}
  3294. +
  3295. +static inline u32 rmap_item_hash_max(struct rmap_item *item, u32 hash)
  3296. +{
  3297. +   u32 hash_max = item->hash_max;
  3298. +
  3299. +   if (!hash_max) {
  3300. +       hash_max = page_hash_max(item->page, hash);
  3301. +
  3302. +       item->hash_max = hash_max;
  3303. +   }
  3304. +
  3305. +   return hash_max;
  3306. +}
  3307. +
  3308. +
  3309. +
  3310. +/**
  3311. + * stable_tree_search() - search the stable tree for a page
  3312. + *
  3313. + * @item:  the rmap_item we are comparing with
  3314. + * @hash:  the hash value of this item->page already calculated
  3315. + *
  3316. + * @return     the page we have found, NULL otherwise. The page returned has
  3317. + *             been gotten.
  3318. + */
  3319. +static struct page *stable_tree_search(struct rmap_item *item, u32 hash)
  3320. +{
  3321. +   struct rb_node *node = root_stable_treep->rb_node;
  3322. +   struct tree_node *tree_node;
  3323. +   unsigned long hash_max;
  3324. +   struct page *page = item->page;
  3325. +   struct stable_node *stable_node;
  3326. +
  3327. +   stable_node = page_stable_node(page);
  3328. +   if (stable_node) {
  3329. +       /* ksm page forked, that is
  3330. +        * if (PageKsm(page) && !in_stable_tree(rmap_item))
  3331. +        * it's actually gotten once outside.
  3332. +        */
  3333. +       get_page(page);
  3334. +       return page;
  3335. +   }
  3336. +
  3337. +   while (node) {
  3338. +       int cmp;
  3339. +
  3340. +       tree_node = rb_entry(node, struct tree_node, node);
  3341. +
  3342. +       cmp = hash_cmp(hash, tree_node->hash);
  3343. +
  3344. +       if (cmp < 0)
  3345. +           node = node->rb_left;
  3346. +       else if (cmp > 0)
  3347. +           node = node->rb_right;
  3348. +       else
  3349. +           break;
  3350. +   }
  3351. +
  3352. +   if (!node)
  3353. +       return NULL;
  3354. +
  3355. +   if (tree_node->count == 1) {
  3356. +       stable_node = rb_entry(tree_node->sub_root.rb_node,
  3357. +                      struct stable_node, node);
  3358. +       BUG_ON(!stable_node);
  3359. +
  3360. +       goto get_page_out;
  3361. +   }
  3362. +
  3363. +   /*
  3364. +    * ok, we have to search the second
  3365. +    * level subtree, hash the page to a
  3366. +    * full strength.
  3367. +    */
  3368. +   node = tree_node->sub_root.rb_node;
  3369. +   BUG_ON(!node);
  3370. +   hash_max = rmap_item_hash_max(item, hash);
  3371. +
  3372. +   while (node) {
  3373. +       int cmp;
  3374. +
  3375. +       stable_node = rb_entry(node, struct stable_node, node);
  3376. +
  3377. +       cmp = hash_cmp(hash_max, stable_node->hash_max);
  3378. +
  3379. +       if (cmp < 0)
  3380. +           node = node->rb_left;
  3381. +       else if (cmp > 0)
  3382. +           node = node->rb_right;
  3383. +       else
  3384. +           goto get_page_out;
  3385. +   }
  3386. +
  3387. +   return NULL;
  3388. +
  3389. +get_page_out:
  3390. +   page = get_uksm_page(stable_node, 1, 1);
  3391. +   return page;
  3392. +}
  3393. +
  3394. +static int try_merge_rmap_item(struct rmap_item *item,
  3395. +                  struct page *kpage,
  3396. +                  struct page *tree_page)
  3397. +{
  3398. +   spinlock_t *ptl;
  3399. +   pte_t *ptep;
  3400. +   unsigned long addr;
  3401. +   struct vm_area_struct *vma = item->slot->vma;
  3402. +
  3403. +   addr = get_rmap_addr(item);
  3404. +   ptep = page_check_address(kpage, vma->vm_mm, addr, &ptl, 0);
  3405. +   if (!ptep)
  3406. +       return 0;
  3407. +
  3408. +   if (pte_write(*ptep)) {
  3409. +       /* has changed, abort! */
  3410. +       pte_unmap_unlock(ptep, ptl);
  3411. +       return 0;
  3412. +   }
  3413. +
  3414. +   get_page(tree_page);
  3415. +   page_add_anon_rmap(tree_page, vma, addr);
  3416. +
  3417. +   flush_cache_page(vma, addr, pte_pfn(*ptep));
  3418. +   ptep_clear_flush(vma, addr, ptep);
  3419. +   set_pte_at_notify(vma->vm_mm, addr, ptep,
  3420. +             mk_pte(tree_page, vma->vm_page_prot));
  3421. +
  3422. +   page_remove_rmap(kpage);
  3423. +   put_page(kpage);
  3424. +
  3425. +   pte_unmap_unlock(ptep, ptl);
  3426. +
  3427. +   return 1;
  3428. +}
  3429. +
  3430. +/**
  3431. + * try_to_merge_with_stable_page() - when two rmap_items need to be inserted
  3432. + * into stable tree, the page was found to be identical to a stable ksm page,
  3433. + * this is the last chance we can merge them into one.
  3434. + *
  3435. + * @item1: the rmap_item holding the page which we wanted to insert
  3436. + *         into stable tree.
  3437. + * @item2: the other rmap_item we found when unstable tree search
  3438. + * @oldpage:   the page currently mapped by the two rmap_items
  3439. + * @tree_page:     the page we found identical in stable tree node
  3440. + * @success1:  return if item1 is successfully merged
  3441. + * @success2:  return if item2 is successfully merged
  3442. + */
  3443. +static void try_merge_with_stable(struct rmap_item *item1,
  3444. +                 struct rmap_item *item2,
  3445. +                 struct page **kpage,
  3446. +                 struct page *tree_page,
  3447. +                 int *success1, int *success2)
  3448. +{
  3449. +   struct vm_area_struct *vma1 = item1->slot->vma;
  3450. +   struct vm_area_struct *vma2 = item2->slot->vma;
  3451. +   *success1 = 0;
  3452. +   *success2 = 0;
  3453. +
  3454. +   if (unlikely(*kpage == tree_page)) {
  3455. +       /* I don't think this can really happen */
  3456. +       printk(KERN_WARNING "UKSM: unexpected condition detected in "
  3457. +           "try_merge_with_stable() -- *kpage == tree_page !\n");
  3458. +       *success1 = 1;
  3459. +       *success2 = 1;
  3460. +       return;
  3461. +   }
  3462. +
  3463. +   if (!PageAnon(*kpage) || !PageKsm(*kpage))
  3464. +       goto failed;
  3465. +
  3466. +   if (!trylock_page(tree_page))
  3467. +       goto failed;
  3468. +
  3469. +   /* If the oldpage is still ksm and still pointed
  3470. +    * to in the right place, and still write protected,
  3471. +    * we are confident it's not changed, no need to
  3472. +    * memcmp anymore.
  3473. +    * be ware, we cannot take nested pte locks,
  3474. +    * deadlock risk.
  3475. +    */
  3476. +   if (!try_merge_rmap_item(item1, *kpage, tree_page))
  3477. +       goto unlock_failed;
  3478. +
  3479. +   /* ok, then vma2, remind that pte1 already set */
  3480. +   if (!try_merge_rmap_item(item2, *kpage, tree_page))
  3481. +       goto success_1;
  3482. +
  3483. +   *success2 = 1;
  3484. +success_1:
  3485. +   *success1 = 1;
  3486. +
  3487. +
  3488. +   if ((*success1 && vma1->vm_flags & VM_LOCKED) ||
  3489. +       (*success2 && vma2->vm_flags & VM_LOCKED)) {
  3490. +       munlock_vma_page(*kpage);
  3491. +       if (!PageMlocked(tree_page))
  3492. +           mlock_vma_page(tree_page);
  3493. +   }
  3494. +
  3495. +   /*
  3496. +    * We do not need oldpage any more in the caller, so can break the lock
  3497. +    * now.
  3498. +    */
  3499. +   unlock_page(*kpage);
  3500. +   *kpage = tree_page; /* Get unlocked outside. */
  3501. +   return;
  3502. +
  3503. +unlock_failed:
  3504. +   unlock_page(tree_page);
  3505. +failed:
  3506. +   return;
  3507. +}
  3508. +
  3509. +static inline void stable_node_hash_max(struct stable_node *node,
  3510. +                    struct page *page, u32 hash)
  3511. +{
  3512. +   u32 hash_max = node->hash_max;
  3513. +
  3514. +   if (!hash_max) {
  3515. +       hash_max = page_hash_max(page, hash);
  3516. +       node->hash_max = hash_max;
  3517. +   }
  3518. +}
  3519. +
  3520. +static inline
  3521. +struct stable_node *new_stable_node(struct tree_node *tree_node,
  3522. +                   struct page *kpage, u32 hash_max)
  3523. +{
  3524. +   struct stable_node *new_stable_node;
  3525. +
  3526. +   new_stable_node = alloc_stable_node();
  3527. +   if (!new_stable_node)
  3528. +       return NULL;
  3529. +
  3530. +   new_stable_node->kpfn = page_to_pfn(kpage);
  3531. +   new_stable_node->hash_max = hash_max;
  3532. +   new_stable_node->tree_node = tree_node;
  3533. +   set_page_stable_node(kpage, new_stable_node);
  3534. +
  3535. +   return new_stable_node;
  3536. +}
  3537. +
  3538. +static inline
  3539. +struct stable_node *first_level_insert(struct tree_node *tree_node,
  3540. +                      struct rmap_item *rmap_item,
  3541. +                      struct rmap_item *tree_rmap_item,
  3542. +                      struct page **kpage, u32 hash,
  3543. +                      int *success1, int *success2)
  3544. +{
  3545. +   int cmp;
  3546. +   struct page *tree_page;
  3547. +   u32 hash_max = 0;
  3548. +   struct stable_node *stable_node, *new_snode;
  3549. +   struct rb_node *parent = NULL, **new;
  3550. +
  3551. +   /* this tree node contains no sub-tree yet */
  3552. +   stable_node = rb_entry(tree_node->sub_root.rb_node,
  3553. +                  struct stable_node, node);
  3554. +
  3555. +   tree_page = get_uksm_page(stable_node, 1, 0);
  3556. +   if (tree_page) {
  3557. +       cmp = memcmp_pages(*kpage, tree_page, 1);
  3558. +       if (!cmp) {
  3559. +           try_merge_with_stable(rmap_item, tree_rmap_item, kpage,
  3560. +                         tree_page, success1, success2);
  3561. +           put_page(tree_page);
  3562. +           if (!*success1 && !*success2)
  3563. +               goto failed;
  3564. +
  3565. +           return stable_node;
  3566. +
  3567. +       } else {
  3568. +           /*
  3569. +            * collision in first level try to create a subtree.
  3570. +            * A new node need to be created.
  3571. +            */
  3572. +           put_page(tree_page);
  3573. +
  3574. +           stable_node_hash_max(stable_node, tree_page,
  3575. +                        tree_node->hash);
  3576. +           hash_max = rmap_item_hash_max(rmap_item, hash);
  3577. +           cmp = hash_cmp(hash_max, stable_node->hash_max);
  3578. +
  3579. +           parent = &stable_node->node;
  3580. +           if (cmp < 0) {
  3581. +               new = &parent->rb_left;
  3582. +           } else if (cmp > 0) {
  3583. +               new = &parent->rb_right;
  3584. +           } else {
  3585. +               goto failed;
  3586. +           }
  3587. +       }
  3588. +
  3589. +   } else {
  3590. +       /* the only stable_node deleted, we reuse its tree_node.
  3591. +        */
  3592. +       parent = NULL;
  3593. +       new = &tree_node->sub_root.rb_node;
  3594. +   }
  3595. +
  3596. +   new_snode = new_stable_node(tree_node, *kpage, hash_max);
  3597. +   if (!new_snode)
  3598. +       goto failed;
  3599. +
  3600. +   rb_link_node(&new_snode->node, parent, new);
  3601. +   rb_insert_color(&new_snode->node, &tree_node->sub_root);
  3602. +   tree_node->count++;
  3603. +   *success1 = *success2 = 1;
  3604. +
  3605. +   return new_snode;
  3606. +
  3607. +failed:
  3608. +   return NULL;
  3609. +}
  3610. +
  3611. +static inline
  3612. +struct stable_node *stable_subtree_insert(struct tree_node *tree_node,
  3613. +                     struct rmap_item *rmap_item,
  3614. +                     struct rmap_item *tree_rmap_item,
  3615. +                     struct page **kpage, u32 hash,
  3616. +                     int *success1, int *success2)
  3617. +{
  3618. +   struct page *tree_page;
  3619. +   u32 hash_max;
  3620. +   struct stable_node *stable_node, *new_snode;
  3621. +   struct rb_node *parent, **new;
  3622. +
  3623. +research:
  3624. +   parent = NULL;
  3625. +   new = &tree_node->sub_root.rb_node;
  3626. +   BUG_ON(!*new);
  3627. +   hash_max = rmap_item_hash_max(rmap_item, hash);
  3628. +   while (*new) {
  3629. +       int cmp;
  3630. +
  3631. +       stable_node = rb_entry(*new, struct stable_node, node);
  3632. +
  3633. +       cmp = hash_cmp(hash_max, stable_node->hash_max);
  3634. +
  3635. +       if (cmp < 0) {
  3636. +           parent = *new;
  3637. +           new = &parent->rb_left;
  3638. +       } else if (cmp > 0) {
  3639. +           parent = *new;
  3640. +           new = &parent->rb_right;
  3641. +       } else {
  3642. +           tree_page = get_uksm_page(stable_node, 1, 0);
  3643. +           if (tree_page) {
  3644. +               cmp = memcmp_pages(*kpage, tree_page, 1);
  3645. +               if (!cmp) {
  3646. +                   try_merge_with_stable(rmap_item,
  3647. +                       tree_rmap_item, kpage,
  3648. +                       tree_page, success1, success2);
  3649. +
  3650. +                   put_page(tree_page);
  3651. +                   if (!*success1 && !*success2)
  3652. +                       goto failed;
  3653. +                   /*
  3654. +                    * successfully merged with a stable
  3655. +                    * node
  3656. +                    */
  3657. +                   return stable_node;
  3658. +               } else {
  3659. +                   put_page(tree_page);
  3660. +                   goto failed;
  3661. +               }
  3662. +           } else {
  3663. +               /*
  3664. +                * stable node may be deleted,
  3665. +                * and subtree maybe
  3666. +                * restructed, cannot
  3667. +                * continue, research it.
  3668. +                */
  3669. +               if (tree_node->count) {
  3670. +                   goto research;
  3671. +               } else {
  3672. +                   /* reuse the tree node*/
  3673. +                   parent = NULL;
  3674. +                   new = &tree_node->sub_root.rb_node;
  3675. +               }
  3676. +           }
  3677. +       }
  3678. +   }
  3679. +
  3680. +   new_snode = new_stable_node(tree_node, *kpage, hash_max);
  3681. +   if (!new_snode)
  3682. +       goto failed;
  3683. +
  3684. +   rb_link_node(&new_snode->node, parent, new);
  3685. +   rb_insert_color(&new_snode->node, &tree_node->sub_root);
  3686. +   tree_node->count++;
  3687. +   *success1 = *success2 = 1;
  3688. +
  3689. +   return new_snode;
  3690. +
  3691. +failed:
  3692. +   return NULL;
  3693. +}
  3694. +
  3695. +
  3696. +/**
  3697. + * stable_tree_insert() - try to insert a merged page in unstable tree to
  3698. + * the stable tree
  3699. + *
  3700. + * @kpage:     the page need to be inserted
  3701. + * @hash:      the current hash of this page
  3702. + * @rmap_item:     the rmap_item being scanned
  3703. + * @tree_rmap_item:    the rmap_item found on unstable tree
  3704. + * @success1:      return if rmap_item is merged
  3705. + * @success2:      return if tree_rmap_item is merged
  3706. + *
  3707. + * @return         the stable_node on stable tree if at least one
  3708. + *             rmap_item is inserted into stable tree, NULL
  3709. + *             otherwise.
  3710. + */
  3711. +static struct stable_node *
  3712. +stable_tree_insert(struct page **kpage, u32 hash,
  3713. +          struct rmap_item *rmap_item,
  3714. +          struct rmap_item *tree_rmap_item,
  3715. +          int *success1, int *success2)
  3716. +{
  3717. +   struct rb_node **new = &root_stable_treep->rb_node;
  3718. +   struct rb_node *parent = NULL;
  3719. +   struct stable_node *stable_node;
  3720. +   struct tree_node *tree_node;
  3721. +   u32 hash_max = 0;
  3722. +
  3723. +   *success1 = *success2 = 0;
  3724. +
  3725. +   while (*new) {
  3726. +       int cmp;
  3727. +
  3728. +       tree_node = rb_entry(*new, struct tree_node, node);
  3729. +
  3730. +       cmp = hash_cmp(hash, tree_node->hash);
  3731. +
  3732. +       if (cmp < 0) {
  3733. +           parent = *new;
  3734. +           new = &parent->rb_left;
  3735. +       } else if (cmp > 0) {
  3736. +           parent = *new;
  3737. +           new = &parent->rb_right;
  3738. +       } else
  3739. +           break;
  3740. +   }
  3741. +
  3742. +   if (*new) {
  3743. +       if (tree_node->count == 1) {
  3744. +           stable_node = first_level_insert(tree_node, rmap_item,
  3745. +                       tree_rmap_item, kpage,
  3746. +                       hash, success1, success2);
  3747. +       } else {
  3748. +           stable_node = stable_subtree_insert(tree_node,
  3749. +                   rmap_item, tree_rmap_item, kpage,
  3750. +                   hash, success1, success2);
  3751. +       }
  3752. +   } else {
  3753. +
  3754. +       /* no tree node found */
  3755. +       tree_node = alloc_tree_node(stable_tree_node_listp);
  3756. +       if (!tree_node) {
  3757. +           stable_node = NULL;
  3758. +           goto out;
  3759. +       }
  3760. +
  3761. +       stable_node = new_stable_node(tree_node, *kpage, hash_max);
  3762. +       if (!stable_node) {
  3763. +           free_tree_node(tree_node);
  3764. +           goto out;
  3765. +       }
  3766. +
  3767. +       tree_node->hash = hash;
  3768. +       rb_link_node(&tree_node->node, parent, new);
  3769. +       rb_insert_color(&tree_node->node, root_stable_treep);
  3770. +       parent = NULL;
  3771. +       new = &tree_node->sub_root.rb_node;
  3772. +
  3773. +       rb_link_node(&stable_node->node, parent, new);
  3774. +       rb_insert_color(&stable_node->node, &tree_node->sub_root);
  3775. +       tree_node->count++;
  3776. +       *success1 = *success2 = 1;
  3777. +   }
  3778. +
  3779. +out:
  3780. +   return stable_node;
  3781. +}
  3782. +
  3783. +
  3784. +/**
  3785. + * get_tree_rmap_item_page() - try to get the page and lock the mmap_sem
  3786. + *
  3787. + * @return     0 on success, -EBUSY if unable to lock the mmap_sem,
  3788. + *             -EINVAL if the page mapping has been changed.
  3789. + */
  3790. +static inline int get_tree_rmap_item_page(struct rmap_item *tree_rmap_item)
  3791. +{
  3792. +   int err;
  3793. +
  3794. +   err = get_mergeable_page_lock_mmap(tree_rmap_item);
  3795. +
  3796. +   if (err == -EINVAL) {
  3797. +       /* its page map has been changed, remove it */
  3798. +       remove_rmap_item_from_tree(tree_rmap_item);
  3799. +   }
  3800. +
  3801. +   /* The page is gotten and mmap_sem is locked now. */
  3802. +   return err;
  3803. +}
  3804. +
  3805. +
  3806. +/**
  3807. + * unstable_tree_search_insert() - search an unstable tree rmap_item with the
  3808. + * same hash value. Get its page and trylock the mmap_sem
  3809. + */
  3810. +static inline
  3811. +struct rmap_item *unstable_tree_search_insert(struct rmap_item *rmap_item,
  3812. +                         u32 hash)
  3813. +
  3814. +{
  3815. +   struct rb_node **new = &root_unstable_tree.rb_node;
  3816. +   struct rb_node *parent = NULL;
  3817. +   struct tree_node *tree_node;
  3818. +   u32 hash_max;
  3819. +   struct rmap_item *tree_rmap_item;
  3820. +
  3821. +   while (*new) {
  3822. +       int cmp;
  3823. +
  3824. +       tree_node = rb_entry(*new, struct tree_node, node);
  3825. +
  3826. +       cmp = hash_cmp(hash, tree_node->hash);
  3827. +
  3828. +       if (cmp < 0) {
  3829. +           parent = *new;
  3830. +           new = &parent->rb_left;
  3831. +       } else if (cmp > 0) {
  3832. +           parent = *new;
  3833. +           new = &parent->rb_right;
  3834. +       } else
  3835. +           break;
  3836. +   }
  3837. +
  3838. +   if (*new) {
  3839. +       /* got the tree_node */
  3840. +       if (tree_node->count == 1) {
  3841. +           tree_rmap_item = rb_entry(tree_node->sub_root.rb_node,
  3842. +                         struct rmap_item, node);
  3843. +           BUG_ON(!tree_rmap_item);
  3844. +
  3845. +           goto get_page_out;
  3846. +       }
  3847. +
  3848. +       /* well, search the collision subtree */
  3849. +       new = &tree_node->sub_root.rb_node;
  3850. +       BUG_ON(!*new);
  3851. +       hash_max = rmap_item_hash_max(rmap_item, hash);
  3852. +
  3853. +       while (*new) {
  3854. +           int cmp;
  3855. +
  3856. +           tree_rmap_item = rb_entry(*new, struct rmap_item,
  3857. +                         node);
  3858. +
  3859. +           cmp = hash_cmp(hash_max, tree_rmap_item->hash_max);
  3860. +           parent = *new;
  3861. +           if (cmp < 0)
  3862. +               new = &parent->rb_left;
  3863. +           else if (cmp > 0)
  3864. +               new = &parent->rb_right;
  3865. +           else
  3866. +               goto get_page_out;
  3867. +       }
  3868. +   } else {
  3869. +       /* alloc a new tree_node */
  3870. +       tree_node = alloc_tree_node(&unstable_tree_node_list);
  3871. +       if (!tree_node)
  3872. +           return NULL;
  3873. +
  3874. +       tree_node->hash = hash;
  3875. +       rb_link_node(&tree_node->node, parent, new);
  3876. +       rb_insert_color(&tree_node->node, &root_unstable_tree);
  3877. +       parent = NULL;
  3878. +       new = &tree_node->sub_root.rb_node;
  3879. +   }
  3880. +
  3881. +   /* did not found even in sub-tree */
  3882. +   rmap_item->tree_node = tree_node;
  3883. +   rmap_item->address |= UNSTABLE_FLAG;
  3884. +   rmap_item->hash_round = uksm_hash_round;
  3885. +   rb_link_node(&rmap_item->node, parent, new);
  3886. +   rb_insert_color(&rmap_item->node, &tree_node->sub_root);
  3887. +
  3888. +   uksm_pages_unshared++;
  3889. +   return NULL;
  3890. +
  3891. +get_page_out:
  3892. +   if (tree_rmap_item->page == rmap_item->page)
  3893. +       return NULL;
  3894. +
  3895. +   if (get_tree_rmap_item_page(tree_rmap_item))
  3896. +       return NULL;
  3897. +
  3898. +   return tree_rmap_item;
  3899. +}
  3900. +
  3901. +static void hold_anon_vma(struct rmap_item *rmap_item,
  3902. +             struct anon_vma *anon_vma)
  3903. +{
  3904. +   rmap_item->anon_vma = anon_vma;
  3905. +   get_anon_vma(anon_vma);
  3906. +}
  3907. +
  3908. +
  3909. +/**
  3910. + * stable_tree_append() - append a rmap_item to a stable node. Deduplication
  3911. + * ratio statistics is done in this function.
  3912. + *
  3913. + */
  3914. +static void stable_tree_append(struct rmap_item *rmap_item,
  3915. +                  struct stable_node *stable_node, int logdedup)
  3916. +{
  3917. +   struct node_vma *node_vma = NULL, *new_node_vma;
  3918. +   struct hlist_node *hlist = NULL, *cont_p = NULL;
  3919. +   unsigned long key = (unsigned long)rmap_item->slot;
  3920. +   unsigned long factor = rmap_item->slot->rung->step;
  3921. +
  3922. +   BUG_ON(!stable_node);
  3923. +   rmap_item->address |= STABLE_FLAG;
  3924. +
  3925. +   if (hlist_empty(&stable_node->hlist)) {
  3926. +       uksm_pages_shared++;
  3927. +       goto node_vma_new;
  3928. +   } else {
  3929. +       uksm_pages_sharing++;
  3930. +   }
  3931. +
  3932. +   hlist_for_each_entry(node_vma, hlist, &stable_node->hlist, hlist) {
  3933. +       if (node_vma->key >= key)
  3934. +           break;
  3935. +
  3936. +       if (logdedup) {
  3937. +           node_vma->slot->pages_bemerged += factor;
  3938. +           if (list_empty(&node_vma->slot->dedup_list))
  3939. +               list_add(&node_vma->slot->dedup_list,
  3940. +                    &vma_slot_dedup);
  3941. +       }
  3942. +   }
  3943. +
  3944. +   if (node_vma) {
  3945. +       if (node_vma->key == key) {
  3946. +           cont_p = hlist->next;
  3947. +           goto node_vma_ok;
  3948. +       } else if (node_vma->key > key) {
  3949. +           cont_p = hlist;
  3950. +       }
  3951. +   }
  3952. +
  3953. +node_vma_new:
  3954. +   /* no same vma already in node, alloc a new node_vma */
  3955. +   new_node_vma = alloc_node_vma();
  3956. +   BUG_ON(!new_node_vma);
  3957. +   new_node_vma->head = stable_node;
  3958. +   new_node_vma->slot = rmap_item->slot;
  3959. +
  3960. +   if (!node_vma) {
  3961. +       hlist_add_head(&new_node_vma->hlist, &stable_node->hlist);
  3962. +   } else if (node_vma->key != key) {
  3963. +       if (node_vma->key < key)
  3964. +           hlist_add_after(&node_vma->hlist, &new_node_vma->hlist);
  3965. +       else {
  3966. +           hlist_add_before(&new_node_vma->hlist,
  3967. +                    &node_vma->hlist);
  3968. +       }
  3969. +
  3970. +   }
  3971. +   node_vma = new_node_vma;
  3972. +
  3973. +node_vma_ok: /* ok, ready to add to the list */
  3974. +   rmap_item->head = node_vma;
  3975. +   hlist_add_head(&rmap_item->hlist, &node_vma->rmap_hlist);
  3976. +   hold_anon_vma(rmap_item, rmap_item->slot->vma->anon_vma);
  3977. +   if (logdedup) {
  3978. +       rmap_item->slot->pages_merged++;
  3979. +       if (cont_p) {
  3980. +           hlist_for_each_entry_continue(node_vma,
  3981. +                             cont_p, hlist) {
  3982. +               node_vma->slot->pages_bemerged += factor;
  3983. +               if (list_empty(&node_vma->slot->dedup_list))
  3984. +                   list_add(&node_vma->slot->dedup_list,
  3985. +                        &vma_slot_dedup);
  3986. +           }
  3987. +       }
  3988. +   }
  3989. +}
  3990. +
  3991. +/*
  3992. + * We use break_ksm to break COW on a ksm page: it's a stripped down
  3993. + *
  3994. + * if (get_user_pages(current, mm, addr, 1, 1, 1, &page, NULL) == 1)
  3995. + *     put_page(page);
  3996. + *
  3997. + * but taking great care only to touch a ksm page, in a VM_MERGEABLE vma,
  3998. + * in case the application has unmapped and remapped mm,addr meanwhile.
  3999. + * Could a ksm page appear anywhere else?  Actually yes, in a VM_PFNMAP
  4000. + * mmap of /dev/mem or /dev/kmem, where we would not want to touch it.
  4001. + */
  4002. +static int break_ksm(struct vm_area_struct *vma, unsigned long addr)
  4003. +{
  4004. +   struct page *page;
  4005. +   int ret = 0;
  4006. +
  4007. +   do {
  4008. +       cond_resched();
  4009. +       page = follow_page(vma, addr, FOLL_GET);
  4010. +       if (IS_ERR_OR_NULL(page))
  4011. +           break;
  4012. +       if (PageKsm(page)) {
  4013. +           ret = handle_mm_fault(vma->vm_mm, vma, addr,
  4014. +                         FAULT_FLAG_WRITE);
  4015. +       } else
  4016. +           ret = VM_FAULT_WRITE;
  4017. +       put_page(page);
  4018. +   } while (!(ret & (VM_FAULT_WRITE | VM_FAULT_SIGBUS | VM_FAULT_OOM)));
  4019. +   /*
  4020. +    * We must loop because handle_mm_fault() may back out if there's
  4021. +    * any difficulty e.g. if pte accessed bit gets updated concurrently.
  4022. +    *
  4023. +    * VM_FAULT_WRITE is what we have been hoping for: it indicates that
  4024. +    * COW has been broken, even if the vma does not permit VM_WRITE;
  4025. +    * but note that a concurrent fault might break PageKsm for us.
  4026. +    *
  4027. +    * VM_FAULT_SIGBUS could occur if we race with truncation of the
  4028. +    * backing file, which also invalidates anonymous pages: that's
  4029. +    * okay, that truncation will have unmapped the PageKsm for us.
  4030. +    *
  4031. +    * VM_FAULT_OOM: at the time of writing (late July 2009), setting
  4032. +    * aside mem_cgroup limits, VM_FAULT_OOM would only be set if the
  4033. +    * current task has TIF_MEMDIE set, and will be OOM killed on return
  4034. +    * to user; and ksmd, having no mm, would never be chosen for that.
  4035. +    *
  4036. +    * But if the mm is in a limited mem_cgroup, then the fault may fail
  4037. +    * with VM_FAULT_OOM even if the current task is not TIF_MEMDIE; and
  4038. +    * even ksmd can fail in this way - though it's usually breaking ksm
  4039. +    * just to undo a merge it made a moment before, so unlikely to oom.
  4040. +    *
  4041. +    * That's a pity: we might therefore have more kernel pages allocated
  4042. +    * than we're counting as nodes in the stable tree; but uksm_do_scan
  4043. +    * will retry to break_cow on each pass, so should recover the page
  4044. +    * in due course.  The important thing is to not let VM_MERGEABLE
  4045. +    * be cleared while any such pages might remain in the area.
  4046. +    */
  4047. +   return (ret & VM_FAULT_OOM) ? -ENOMEM : 0;
  4048. +}
  4049. +
  4050. +static void break_cow(struct rmap_item *rmap_item)
  4051. +{
  4052. +   struct vm_area_struct *vma = rmap_item->slot->vma;
  4053. +   struct mm_struct *mm = vma->vm_mm;
  4054. +   unsigned long addr = get_rmap_addr(rmap_item);
  4055. +
  4056. +   if (uksm_test_exit(mm))
  4057. +       goto out;
  4058. +
  4059. +   break_ksm(vma, addr);
  4060. +out:
  4061. +   return;
  4062. +}
  4063. +
  4064. +/*
  4065. + * Though it's very tempting to unmerge in_stable_tree(rmap_item)s rather
  4066. + * than check every pte of a given vma, the locking doesn't quite work for
  4067. + * that - an rmap_item is assigned to the stable tree after inserting ksm
  4068. + * page and upping mmap_sem.  Nor does it fit with the way we skip dup'ing
  4069. + * rmap_items from parent to child at fork time (so as not to waste time
  4070. + * if exit comes before the next scan reaches it).
  4071. + *
  4072. + * Similarly, although we'd like to remove rmap_items (so updating counts
  4073. + * and freeing memory) when unmerging an area, it's easier to leave that
  4074. + * to the next pass of ksmd - consider, for example, how ksmd might be
  4075. + * in cmp_and_merge_page on one of the rmap_items we would be removing.
  4076. + */
  4077. +inline int unmerge_uksm_pages(struct vm_area_struct *vma,
  4078. +             unsigned long start, unsigned long end)
  4079. +{
  4080. +   unsigned long addr;
  4081. +   int err = 0;
  4082. +
  4083. +   for (addr = start; addr < end && !err; addr += PAGE_SIZE) {
  4084. +       if (uksm_test_exit(vma->vm_mm))
  4085. +           break;
  4086. +       if (signal_pending(current))
  4087. +           err = -ERESTARTSYS;
  4088. +       else
  4089. +           err = break_ksm(vma, addr);
  4090. +   }
  4091. +   return err;
  4092. +}
  4093. +
  4094. +static inline void inc_uksm_pages_scanned(void)
  4095. +{
  4096. +   u64 delta;
  4097. +
  4098. +
  4099. +   if (uksm_pages_scanned == U64_MAX) {
  4100. +       encode_benefit();
  4101. +
  4102. +       delta = uksm_pages_scanned >> pages_scanned_base;
  4103. +
  4104. +       if (CAN_OVERFLOW_U64(pages_scanned_stored, delta)) {
  4105. +           pages_scanned_stored >>= 1;
  4106. +           delta >>= 1;
  4107. +           pages_scanned_base++;
  4108. +       }
  4109. +
  4110. +       pages_scanned_stored += delta;
  4111. +
  4112. +       uksm_pages_scanned = uksm_pages_scanned_last = 0;
  4113. +   }
  4114. +
  4115. +   uksm_pages_scanned++;
  4116. +}
  4117. +
  4118. +static inline int find_zero_page_hash(int strength, u32 hash)
  4119. +{
  4120. +   return (zero_hash_table[strength] == hash);
  4121. +}
  4122. +
  4123. +static
  4124. +int cmp_and_merge_zero_page(struct vm_area_struct *vma, struct page *page)
  4125. +{
  4126. +   struct page *zero_page = empty_uksm_zero_page;
  4127. +   struct mm_struct *mm = vma->vm_mm;
  4128. +   pte_t orig_pte = __pte(0);
  4129. +   int err = -EFAULT;
  4130. +
  4131. +   if (uksm_test_exit(mm))
  4132. +       goto out;
  4133. +
  4134. +   if (PageTransCompound(page) && page_trans_compound_anon_split(page))
  4135. +       goto out;
  4136. +   BUG_ON(PageTransCompound(page));
  4137. +
  4138. +   if (!PageAnon(page))
  4139. +       goto out;
  4140. +
  4141. +   if (!trylock_page(page))
  4142. +       goto out;
  4143. +
  4144. +   if (write_protect_page(vma, page, &orig_pte, 0) == 0) {
  4145. +       if (is_page_full_zero(page))
  4146. +           err = replace_page(vma, page, zero_page, orig_pte);
  4147. +   }
  4148. +
  4149. +   unlock_page(page);
  4150. +out:
  4151. +   return err;
  4152. +}
  4153. +
  4154. +/*
  4155. + * cmp_and_merge_page() - first see if page can be merged into the stable
  4156. + * tree; if not, compare hash to previous and if it's the same, see if page
  4157. + * can be inserted into the unstable tree, or merged with a page already there
  4158. + * and both transferred to the stable tree.
  4159. + *
  4160. + * @page: the page that we are searching identical page to.
  4161. + * @rmap_item: the reverse mapping into the virtual address of this page
  4162. + */
  4163. +static void cmp_and_merge_page(struct rmap_item *rmap_item, u32 hash)
  4164. +{
  4165. +   struct rmap_item *tree_rmap_item;
  4166. +   struct page *page;
  4167. +   struct page *kpage = NULL;
  4168. +   u32 hash_max;
  4169. +   int err;
  4170. +   unsigned int success1, success2;
  4171. +   struct stable_node *snode;
  4172. +   int cmp;
  4173. +   struct rb_node *parent = NULL, **new;
  4174. +
  4175. +   remove_rmap_item_from_tree(rmap_item);
  4176. +   page = rmap_item->page;
  4177. +
  4178. +   /* We first start with searching the page inside the stable tree */
  4179. +   kpage = stable_tree_search(rmap_item, hash);
  4180. +   if (kpage) {
  4181. +       err = try_to_merge_with_uksm_page(rmap_item, kpage,
  4182. +                        hash);
  4183. +       if (!err) {
  4184. +           /*
  4185. +            * The page was successfully merged, add
  4186. +            * its rmap_item to the stable tree.
  4187. +            * page lock is needed because it's
  4188. +            * racing with try_to_unmap_ksm(), etc.
  4189. +            */
  4190. +           lock_page(kpage);
  4191. +           snode = page_stable_node(kpage);
  4192. +           stable_tree_append(rmap_item, snode, 1);
  4193. +           unlock_page(kpage);
  4194. +           put_page(kpage);
  4195. +           return; /* success */
  4196. +       }
  4197. +       put_page(kpage);
  4198. +
  4199. +       /*
  4200. +        * if it's a collision and it has been search in sub-rbtree
  4201. +        * (hash_max != 0), we want to abort, because if it is
  4202. +        * successfully merged in unstable tree, the collision trends to
  4203. +        * happen again.
  4204. +        */
  4205. +       if (err == MERGE_ERR_COLLI && rmap_item->hash_max)
  4206. +           return;
  4207. +   }
  4208. +
  4209. +   tree_rmap_item =
  4210. +       unstable_tree_search_insert(rmap_item, hash);
  4211. +   if (tree_rmap_item) {
  4212. +       err = try_to_merge_two_pages(rmap_item, tree_rmap_item, hash);
  4213. +       /*
  4214. +        * As soon as we merge this page, we want to remove the
  4215. +        * rmap_item of the page we have merged with from the unstable
  4216. +        * tree, and insert it instead as new node in the stable tree.
  4217. +        */
  4218. +       if (!err) {
  4219. +           kpage = page;
  4220. +           remove_rmap_item_from_tree(tree_rmap_item);
  4221. +           lock_page(kpage);
  4222. +           snode = stable_tree_insert(&kpage, hash,
  4223. +                          rmap_item, tree_rmap_item,
  4224. +                          &success1, &success2);
  4225. +
  4226. +           /*
  4227. +            * Do not log dedup for tree item, it's not counted as
  4228. +            * scanned in this round.
  4229. +            */
  4230. +           if (success2)
  4231. +               stable_tree_append(tree_rmap_item, snode, 0);
  4232. +
  4233. +           /*
  4234. +            * The order of these two stable append is important:
  4235. +            * we are scanning rmap_item.
  4236. +            */
  4237. +           if (success1)
  4238. +               stable_tree_append(rmap_item, snode, 1);
  4239. +
  4240. +           /*
  4241. +            * The original kpage may be unlocked inside
  4242. +            * stable_tree_insert() already. This page
  4243. +            * should be unlocked before doing
  4244. +            * break_cow().
  4245. +            */
  4246. +           unlock_page(kpage);
  4247. +
  4248. +           if (!success1)
  4249. +               break_cow(rmap_item);
  4250. +
  4251. +           if (!success2)
  4252. +               break_cow(tree_rmap_item);
  4253. +
  4254. +       } else if (err == MERGE_ERR_COLLI) {
  4255. +           BUG_ON(tree_rmap_item->tree_node->count > 1);
  4256. +
  4257. +           rmap_item_hash_max(tree_rmap_item,
  4258. +                      tree_rmap_item->tree_node->hash);
  4259. +
  4260. +           hash_max = rmap_item_hash_max(rmap_item, hash);
  4261. +           cmp = hash_cmp(hash_max, tree_rmap_item->hash_max);
  4262. +           parent = &tree_rmap_item->node;
  4263. +           if (cmp < 0)
  4264. +               new = &parent->rb_left;
  4265. +           else if (cmp > 0)
  4266. +               new = &parent->rb_right;
  4267. +           else
  4268. +               goto put_up_out;
  4269. +
  4270. +           rmap_item->tree_node = tree_rmap_item->tree_node;
  4271. +           rmap_item->address |= UNSTABLE_FLAG;
  4272. +           rmap_item->hash_round = uksm_hash_round;
  4273. +           rb_link_node(&rmap_item->node, parent, new);
  4274. +           rb_insert_color(&rmap_item->node,
  4275. +                   &tree_rmap_item->tree_node->sub_root);
  4276. +           rmap_item->tree_node->count++;
  4277. +       } else {
  4278. +           /*
  4279. +            * either one of the page has changed or they collide
  4280. +            * at the max hash, we consider them as ill items.
  4281. +            */
  4282. +           remove_rmap_item_from_tree(tree_rmap_item);
  4283. +       }
  4284. +put_up_out:
  4285. +       put_page(tree_rmap_item->page);
  4286. +       up_read(&tree_rmap_item->slot->vma->vm_mm->mmap_sem);
  4287. +   }
  4288. +}
  4289. +
  4290. +
  4291. +
  4292. +
  4293. +static inline unsigned long get_pool_index(struct vma_slot *slot,
  4294. +                      unsigned long index)
  4295. +{
  4296. +   unsigned long pool_index;
  4297. +
  4298. +   pool_index = (sizeof(struct rmap_list_entry *) * index) >> PAGE_SHIFT;
  4299. +   if (pool_index >= slot->pool_size)
  4300. +       BUG();
  4301. +   return pool_index;
  4302. +}
  4303. +
  4304. +static inline unsigned long index_page_offset(unsigned long index)
  4305. +{
  4306. +   return offset_in_page(sizeof(struct rmap_list_entry *) * index);
  4307. +}
  4308. +
  4309. +static inline
  4310. +struct rmap_list_entry *get_rmap_list_entry(struct vma_slot *slot,
  4311. +                       unsigned long index, int need_alloc)
  4312. +{
  4313. +   unsigned long pool_index;
  4314. +   struct page *page;
  4315. +   void *addr;
  4316. +
  4317. +
  4318. +   pool_index = get_pool_index(slot, index);
  4319. +   if (!slot->rmap_list_pool[pool_index]) {
  4320. +       if (!need_alloc)
  4321. +           return NULL;
  4322. +
  4323. +       page = alloc_page(GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN);
  4324. +       if (!page)
  4325. +           return NULL;
  4326. +
  4327. +       slot->rmap_list_pool[pool_index] = page;
  4328. +   }
  4329. +
  4330. +   addr = kmap(slot->rmap_list_pool[pool_index]);
  4331. +   addr += index_page_offset(index);
  4332. +
  4333. +   return addr;
  4334. +}
  4335. +
  4336. +static inline void put_rmap_list_entry(struct vma_slot *slot,
  4337. +                      unsigned long index)
  4338. +{
  4339. +   unsigned long pool_index;
  4340. +
  4341. +   pool_index = get_pool_index(slot, index);
  4342. +   BUG_ON(!slot->rmap_list_pool[pool_index]);
  4343. +   kunmap(slot->rmap_list_pool[pool_index]);
  4344. +}
  4345. +
  4346. +static inline int entry_is_new(struct rmap_list_entry *entry)
  4347. +{
  4348. +   return !entry->item;
  4349. +}
  4350. +
  4351. +static inline unsigned long get_index_orig_addr(struct vma_slot *slot,
  4352. +                       unsigned long index)
  4353. +{
  4354. +   return slot->vma->vm_start + (index << PAGE_SHIFT);
  4355. +}
  4356. +
  4357. +static inline unsigned long get_entry_address(struct rmap_list_entry *entry)
  4358. +{
  4359. +   unsigned long addr;
  4360. +
  4361. +   if (is_addr(entry->addr))
  4362. +       addr = get_clean_addr(entry->addr);
  4363. +   else if (entry->item)
  4364. +       addr = get_rmap_addr(entry->item);
  4365. +   else
  4366. +       BUG();
  4367. +
  4368. +   return addr;
  4369. +}
  4370. +
  4371. +static inline struct rmap_item *get_entry_item(struct rmap_list_entry *entry)
  4372. +{
  4373. +   if (is_addr(entry->addr))
  4374. +       return NULL;
  4375. +
  4376. +   return entry->item;
  4377. +}
  4378. +
  4379. +static inline void inc_rmap_list_pool_count(struct vma_slot *slot,
  4380. +                       unsigned long index)
  4381. +{
  4382. +   unsigned long pool_index;
  4383. +
  4384. +   pool_index = get_pool_index(slot, index);
  4385. +   BUG_ON(!slot->rmap_list_pool[pool_index]);
  4386. +   slot->pool_counts[pool_index]++;
  4387. +}
  4388. +
  4389. +static inline void dec_rmap_list_pool_count(struct vma_slot *slot,
  4390. +                       unsigned long index)
  4391. +{
  4392. +   unsigned long pool_index;
  4393. +
  4394. +   pool_index = get_pool_index(slot, index);
  4395. +   BUG_ON(!slot->rmap_list_pool[pool_index]);
  4396. +   BUG_ON(!slot->pool_counts[pool_index]);
  4397. +   slot->pool_counts[pool_index]--;
  4398. +}
  4399. +
  4400. +static inline int entry_has_rmap(struct rmap_list_entry *entry)
  4401. +{
  4402. +   return !is_addr(entry->addr) && entry->item;
  4403. +}
  4404. +
  4405. +static inline void swap_entries(struct rmap_list_entry *entry1,
  4406. +               unsigned long index1,
  4407. +               struct rmap_list_entry *entry2,
  4408. +               unsigned long index2)
  4409. +{
  4410. +   struct rmap_list_entry tmp;
  4411. +
  4412. +   /* swapping two new entries is meaningless */
  4413. +   BUG_ON(entry_is_new(entry1) && entry_is_new(entry2));
  4414. +
  4415. +   tmp = *entry1;
  4416. +   *entry1 = *entry2;
  4417. +   *entry2 = tmp;
  4418. +
  4419. +   if (entry_has_rmap(entry1))
  4420. +       entry1->item->entry_index = index1;
  4421. +
  4422. +   if (entry_has_rmap(entry2))
  4423. +       entry2->item->entry_index = index2;
  4424. +
  4425. +   if (entry_has_rmap(entry1) && !entry_has_rmap(entry2)) {
  4426. +       inc_rmap_list_pool_count(entry1->item->slot, index1);
  4427. +       dec_rmap_list_pool_count(entry1->item->slot, index2);
  4428. +   } else if (!entry_has_rmap(entry1) && entry_has_rmap(entry2)) {
  4429. +       inc_rmap_list_pool_count(entry2->item->slot, index2);
  4430. +       dec_rmap_list_pool_count(entry2->item->slot, index1);
  4431. +   }
  4432. +}
  4433. +
  4434. +static inline void free_entry_item(struct rmap_list_entry *entry)
  4435. +{
  4436. +   unsigned long index;
  4437. +   struct rmap_item *item;
  4438. +
  4439. +   if (!is_addr(entry->addr)) {
  4440. +       BUG_ON(!entry->item);
  4441. +       item = entry->item;
  4442. +       entry->addr = get_rmap_addr(item);
  4443. +       set_is_addr(entry->addr);
  4444. +       index = item->entry_index;
  4445. +       remove_rmap_item_from_tree(item);
  4446. +       dec_rmap_list_pool_count(item->slot, index);
  4447. +       free_rmap_item(item);
  4448. +   }
  4449. +}
  4450. +
  4451. +static inline int pool_entry_boundary(unsigned long index)
  4452. +{
  4453. +   unsigned long linear_addr;
  4454. +
  4455. +   linear_addr = sizeof(struct rmap_list_entry *) * index;
  4456. +   return index && !offset_in_page(linear_addr);
  4457. +}
  4458. +
  4459. +static inline void try_free_last_pool(struct vma_slot *slot,
  4460. +                     unsigned long index)
  4461. +{
  4462. +   unsigned long pool_index;
  4463. +
  4464. +   pool_index = get_pool_index(slot, index);
  4465. +   if (slot->rmap_list_pool[pool_index] &&
  4466. +       !slot->pool_counts[pool_index]) {
  4467. +       __free_page(slot->rmap_list_pool[pool_index]);
  4468. +       slot->rmap_list_pool[pool_index] = NULL;
  4469. +       slot->flags |= UKSM_SLOT_NEED_SORT;
  4470. +   }
  4471. +
  4472. +}
  4473. +
  4474. +static inline unsigned long vma_item_index(struct vm_area_struct *vma,
  4475. +                      struct rmap_item *item)
  4476. +{
  4477. +   return (get_rmap_addr(item) - vma->vm_start) >> PAGE_SHIFT;
  4478. +}
  4479. +
  4480. +static int within_same_pool(struct vma_slot *slot,
  4481. +               unsigned long i, unsigned long j)
  4482. +{
  4483. +   unsigned long pool_i, pool_j;
  4484. +
  4485. +   pool_i = get_pool_index(slot, i);
  4486. +   pool_j = get_pool_index(slot, j);
  4487. +
  4488. +   return (pool_i == pool_j);
  4489. +}
  4490. +
  4491. +static void sort_rmap_entry_list(struct vma_slot *slot)
  4492. +{
  4493. +   unsigned long i, j;
  4494. +   struct rmap_list_entry *entry, *swap_entry;
  4495. +
  4496. +   entry = get_rmap_list_entry(slot, 0, 0);
  4497. +   for (i = 0; i < slot->pages; ) {
  4498. +
  4499. +       if (!entry)
  4500. +           goto skip_whole_pool;
  4501. +
  4502. +       if (entry_is_new(entry))
  4503. +           goto next_entry;
  4504. +
  4505. +       if (is_addr(entry->addr)) {
  4506. +           entry->addr = 0;
  4507. +           goto next_entry;
  4508. +       }
  4509. +
  4510. +       j = vma_item_index(slot->vma, entry->item);
  4511. +       if (j == i)
  4512. +           goto next_entry;
  4513. +
  4514. +       if (within_same_pool(slot, i, j))
  4515. +           swap_entry = entry + j - i;
  4516. +       else
  4517. +           swap_entry = get_rmap_list_entry(slot, j, 1);
  4518. +
  4519. +       swap_entries(entry, i, swap_entry, j);
  4520. +       if (!within_same_pool(slot, i, j))
  4521. +           put_rmap_list_entry(slot, j);
  4522. +       continue;
  4523. +
  4524. +skip_whole_pool:
  4525. +       i += PAGE_SIZE / sizeof(*entry);
  4526. +       if (i < slot->pages)
  4527. +           entry = get_rmap_list_entry(slot, i, 0);
  4528. +       continue;
  4529. +
  4530. +next_entry:
  4531. +       if (i >= slot->pages - 1 ||
  4532. +           !within_same_pool(slot, i, i + 1)) {
  4533. +           put_rmap_list_entry(slot, i);
  4534. +           if (i + 1 < slot->pages)
  4535. +               entry = get_rmap_list_entry(slot, i + 1, 0);
  4536. +       } else
  4537. +           entry++;
  4538. +       i++;
  4539. +       continue;
  4540. +   }
  4541. +
  4542. +   /* free empty pool entries which contain no rmap_item */
  4543. +   /* CAN be simplied to based on only pool_counts when bug freed !!!!! */
  4544. +   for (i = 0; i < slot->pool_size; i++) {
  4545. +       unsigned char has_rmap;
  4546. +       void *addr;
  4547. +
  4548. +       if (!slot->rmap_list_pool[i])
  4549. +           continue;
  4550. +
  4551. +       has_rmap = 0;
  4552. +       addr = kmap(slot->rmap_list_pool[i]);
  4553. +       BUG_ON(!addr);
  4554. +       for (j = 0; j < PAGE_SIZE / sizeof(*entry); j++) {
  4555. +           entry = (struct rmap_list_entry *)addr + j;
  4556. +           if (is_addr(entry->addr))
  4557. +               continue;
  4558. +           if (!entry->item)
  4559. +               continue;
  4560. +           has_rmap = 1;
  4561. +       }
  4562. +       kunmap(slot->rmap_list_pool[i]);
  4563. +       if (!has_rmap) {
  4564. +           BUG_ON(slot->pool_counts[i]);
  4565. +           __free_page(slot->rmap_list_pool[i]);
  4566. +           slot->rmap_list_pool[i] = NULL;
  4567. +       }
  4568. +   }
  4569. +
  4570. +   slot->flags &= ~UKSM_SLOT_NEED_SORT;
  4571. +}
  4572. +
  4573. +/*
  4574. + * vma_fully_scanned() - if all the pages in this slot have been scanned.
  4575. + */
  4576. +static inline int vma_fully_scanned(struct vma_slot *slot)
  4577. +{
  4578. +   return slot->pages_scanned == slot->pages;
  4579. +}
  4580. +
  4581. +/**
  4582. + * get_next_rmap_item() - Get the next rmap_item in a vma_slot according to
  4583. + * its random permutation. This function is embedded with the random
  4584. + * permutation index management code.
  4585. + */
  4586. +static struct rmap_item *get_next_rmap_item(struct vma_slot *slot, u32 *hash)
  4587. +{
  4588. +   unsigned long rand_range, addr, swap_index, scan_index;
  4589. +   struct rmap_item *item = NULL;
  4590. +   struct rmap_list_entry *scan_entry, *swap_entry = NULL;
  4591. +   struct page *page;
  4592. +
  4593. +   scan_index = swap_index = slot->pages_scanned % slot->pages;
  4594. +
  4595. +   if (pool_entry_boundary(scan_index))
  4596. +       try_free_last_pool(slot, scan_index - 1);
  4597. +
  4598. +   if (vma_fully_scanned(slot)) {
  4599. +       if (slot->flags & UKSM_SLOT_NEED_SORT)
  4600. +           slot->flags |= UKSM_SLOT_NEED_RERAND;
  4601. +       else
  4602. +           slot->flags &= ~UKSM_SLOT_NEED_RERAND;
  4603. +       if (slot->flags & UKSM_SLOT_NEED_SORT)
  4604. +           sort_rmap_entry_list(slot);
  4605. +   }
  4606. +
  4607. +   scan_entry = get_rmap_list_entry(slot, scan_index, 1);
  4608. +   if (!scan_entry)
  4609. +       return NULL;
  4610. +
  4611. +   if (entry_is_new(scan_entry)) {
  4612. +       scan_entry->addr = get_index_orig_addr(slot, scan_index);
  4613. +       set_is_addr(scan_entry->addr);
  4614. +   }
  4615. +
  4616. +   if (slot->flags & UKSM_SLOT_NEED_RERAND) {
  4617. +       rand_range = slot->pages - scan_index;
  4618. +       BUG_ON(!rand_range);
  4619. +       swap_index = scan_index + (random32() % rand_range);
  4620. +   }
  4621. +
  4622. +   if (swap_index != scan_index) {
  4623. +       swap_entry = get_rmap_list_entry(slot, swap_index, 1);
  4624. +       if (entry_is_new(swap_entry)) {
  4625. +           swap_entry->addr = get_index_orig_addr(slot,
  4626. +                                  swap_index);
  4627. +           set_is_addr(swap_entry->addr);
  4628. +       }
  4629. +       swap_entries(scan_entry, scan_index, swap_entry, swap_index);
  4630. +   }
  4631. +
  4632. +   addr = get_entry_address(scan_entry);
  4633. +   item = get_entry_item(scan_entry);
  4634. +   BUG_ON(addr > slot->vma->vm_end || addr < slot->vma->vm_start);
  4635. +
  4636. +   page = follow_page(slot->vma, addr, FOLL_GET);
  4637. +   if (IS_ERR_OR_NULL(page))
  4638. +       goto nopage;
  4639. +
  4640. +   if (!PageAnon(page) && !page_trans_compound_anon(page))
  4641. +       goto putpage;
  4642. +
  4643. +   /*check is zero_page pfn or uksm_zero_page*/
  4644. +   if ((page_to_pfn(page) == zero_pfn)
  4645. +           || (page_to_pfn(page) == uksm_zero_pfn))
  4646. +       goto putpage;
  4647. +
  4648. +   flush_anon_page(slot->vma, page, addr);
  4649. +   flush_dcache_page(page);
  4650. +
  4651. +
  4652. +   *hash = page_hash(page, hash_strength, 1);
  4653. +   inc_uksm_pages_scanned();
  4654. +   /*if the page content all zero, re-map to zero-page*/
  4655. +   if (find_zero_page_hash(hash_strength, *hash)) {
  4656. +       if (!cmp_and_merge_zero_page(slot->vma, page)) {
  4657. +           slot->pages_merged++;
  4658. +           inc_zone_page_state(page, NR_UKSM_ZERO_PAGES);
  4659. +           dec_mm_counter(slot->mm, MM_ANONPAGES);
  4660. +
  4661. +           /* For full-zero pages, no need to create rmap item */
  4662. +           goto putpage;
  4663. +       } else {
  4664. +           inc_rshash_neg(memcmp_cost / 2);
  4665. +       }
  4666. +   }
  4667. +
  4668. +   if (!item) {
  4669. +       item = alloc_rmap_item();
  4670. +       if (item) {
  4671. +           /* It has already been zeroed */
  4672. +           item->slot = slot;
  4673. +           item->address = addr;
  4674. +           item->entry_index = scan_index;
  4675. +           scan_entry->item = item;
  4676. +           inc_rmap_list_pool_count(slot, scan_index);
  4677. +       } else
  4678. +           goto putpage;
  4679. +   }
  4680. +
  4681. +   BUG_ON(item->slot != slot);
  4682. +   /* the page may have changed */
  4683. +   item->page = page;
  4684. +   put_rmap_list_entry(slot, scan_index);
  4685. +   if (swap_entry)
  4686. +       put_rmap_list_entry(slot, swap_index);
  4687. +   return item;
  4688. +
  4689. +putpage:
  4690. +   put_page(page);
  4691. +   page = NULL;
  4692. +nopage:
  4693. +   /* no page, store addr back and free rmap_item if possible */
  4694. +   free_entry_item(scan_entry);
  4695. +   put_rmap_list_entry(slot, scan_index);
  4696. +   if (swap_entry)
  4697. +       put_rmap_list_entry(slot, swap_index);
  4698. +   return NULL;
  4699. +}
  4700. +
  4701. +static inline int in_stable_tree(struct rmap_item *rmap_item)
  4702. +{
  4703. +   return rmap_item->address & STABLE_FLAG;
  4704. +}
  4705. +
  4706. +/**
  4707. + * scan_vma_one_page() - scan the next page in a vma_slot. Called with
  4708. + * mmap_sem locked.
  4709. + */
  4710. +static noinline void scan_vma_one_page(struct vma_slot *slot)
  4711. +{
  4712. +   u32 hash;
  4713. +   struct mm_struct *mm;
  4714. +   struct rmap_item *rmap_item = NULL;
  4715. +   struct vm_area_struct *vma = slot->vma;
  4716. +
  4717. +   mm = vma->vm_mm;
  4718. +   BUG_ON(!mm);
  4719. +   BUG_ON(!slot);
  4720. +
  4721. +   rmap_item = get_next_rmap_item(slot, &hash);
  4722. +   if (!rmap_item)
  4723. +       goto out1;
  4724. +
  4725. +   if (PageKsm(rmap_item->page) && in_stable_tree(rmap_item))
  4726. +       goto out2;
  4727. +
  4728. +   cmp_and_merge_page(rmap_item, hash);
  4729. +out2:
  4730. +   put_page(rmap_item->page);
  4731. +out1:
  4732. +   slot->pages_scanned++;
  4733. +   if (slot->fully_scanned_round != fully_scanned_round)
  4734. +       scanned_virtual_pages++;
  4735. +
  4736. +   if (vma_fully_scanned(slot))
  4737. +       slot->fully_scanned_round = fully_scanned_round;
  4738. +}
  4739. +
  4740. +static inline unsigned long rung_get_pages(struct scan_rung *rung)
  4741. +{
  4742. +   struct slot_tree_node *node;
  4743. +
  4744. +   if (!rung->vma_root.rnode)
  4745. +       return 0;
  4746. +
  4747. +   node = container_of(rung->vma_root.rnode, struct slot_tree_node, snode);
  4748. +
  4749. +   return node->size;
  4750. +}
  4751. +
  4752. +#define RUNG_SAMPLED_MIN   3
  4753. +
  4754. +static inline
  4755. +void uksm_calc_rung_step(struct scan_rung *rung,
  4756. +            unsigned long page_time, unsigned long ratio)
  4757. +{
  4758. +   unsigned long sampled, pages;
  4759. +
  4760. +   /* will be fully scanned ? */
  4761. +   if (!rung->cover_msecs) {
  4762. +       rung->step = 1;
  4763. +       return;
  4764. +   }
  4765. +
  4766. +   sampled = rung->cover_msecs * (NSEC_PER_MSEC / TIME_RATIO_SCALE)
  4767. +         * ratio / page_time;
  4768. +
  4769. +   /*
  4770. +    *  Before we finsish a scan round and expensive per-round jobs,
  4771. +    *  we need to have a chance to estimate the per page time. So
  4772. +    *  the sampled number can not be too small.
  4773. +    */
  4774. +   if (sampled < RUNG_SAMPLED_MIN)
  4775. +       sampled = RUNG_SAMPLED_MIN;
  4776. +
  4777. +   pages = rung_get_pages(rung);
  4778. +   if (likely(pages > sampled))
  4779. +       rung->step = pages / sampled;
  4780. +   else
  4781. +       rung->step = 1;
  4782. +}
  4783. +
  4784. +static inline int step_need_recalc(struct scan_rung *rung)
  4785. +{
  4786. +   unsigned long pages, stepmax;
  4787. +
  4788. +   pages = rung_get_pages(rung);
  4789. +   stepmax = pages / RUNG_SAMPLED_MIN;
  4790. +
  4791. +   return pages && (rung->step > pages ||
  4792. +            (stepmax && rung->step > stepmax));
  4793. +}
  4794. +
  4795. +static inline
  4796. +void reset_current_scan(struct scan_rung *rung, int finished, int step_recalc)
  4797. +{
  4798. +   struct vma_slot *slot;
  4799. +
  4800. +   if (finished)
  4801. +       rung->flags |= UKSM_RUNG_ROUND_FINISHED;
  4802. +
  4803. +   if (step_recalc || step_need_recalc(rung)) {
  4804. +       uksm_calc_rung_step(rung, uksm_ema_page_time, rung->cpu_ratio);
  4805. +       BUG_ON(step_need_recalc(rung));
  4806. +   }
  4807. +
  4808. +   slot_iter_index = random32() % rung->step;
  4809. +   BUG_ON(!rung->vma_root.rnode);
  4810. +   slot = sradix_tree_next(&rung->vma_root, NULL, 0, slot_iter);
  4811. +   BUG_ON(!slot);
  4812. +
  4813. +   rung->current_scan = slot;
  4814. +   rung->current_offset = slot_iter_index;
  4815. +}
  4816. +
  4817. +static inline struct sradix_tree_root *slot_get_root(struct vma_slot *slot)
  4818. +{
  4819. +   return &slot->rung->vma_root;
  4820. +}
  4821. +
  4822. +/*
  4823. + * return if resetted.
  4824. + */
  4825. +static int advance_current_scan(struct scan_rung *rung)
  4826. +{
  4827. +   unsigned short n;
  4828. +   struct vma_slot *slot, *next = NULL;
  4829. +
  4830. +   BUG_ON(!rung->vma_root.num);
  4831. +
  4832. +   slot = rung->current_scan;
  4833. +   n = (slot->pages - rung->current_offset) % rung->step;
  4834. +   slot_iter_index = rung->step - n;
  4835. +   next = sradix_tree_next(&rung->vma_root, slot->snode,
  4836. +               slot->sindex, slot_iter);
  4837. +
  4838. +   if (next) {
  4839. +       rung->current_offset = slot_iter_index;
  4840. +       rung->current_scan = next;
  4841. +       return 0;
  4842. +   } else {
  4843. +       reset_current_scan(rung, 1, 0);
  4844. +       return 1;
  4845. +   }
  4846. +}
  4847. +
  4848. +static inline void rung_rm_slot(struct vma_slot *slot)
  4849. +{
  4850. +   struct scan_rung *rung = slot->rung;
  4851. +   struct sradix_tree_root *root;
  4852. +
  4853. +   if (rung->current_scan == slot)
  4854. +       advance_current_scan(rung);
  4855. +
  4856. +   root = slot_get_root(slot);
  4857. +   sradix_tree_delete_from_leaf(root, slot->snode, slot->sindex);
  4858. +   slot->snode = NULL;
  4859. +   if (step_need_recalc(rung)) {
  4860. +       uksm_calc_rung_step(rung, uksm_ema_page_time, rung->cpu_ratio);
  4861. +       BUG_ON(step_need_recalc(rung));
  4862. +   }
  4863. +
  4864. +   /* In case advance_current_scan loop back to this slot again */
  4865. +   if (rung->vma_root.num && rung->current_scan == slot)
  4866. +       reset_current_scan(slot->rung, 1, 0);
  4867. +}
  4868. +
  4869. +static inline void rung_add_new_slots(struct scan_rung *rung,
  4870. +           struct vma_slot **slots, unsigned long num)
  4871. +{
  4872. +   int err;
  4873. +   struct vma_slot *slot;
  4874. +   unsigned long i;
  4875. +   struct sradix_tree_root *root = &rung->vma_root;
  4876. +
  4877. +   err = sradix_tree_enter(root, (void **)slots, num);
  4878. +   BUG_ON(err);
  4879. +
  4880. +   for (i = 0; i < num; i++) {
  4881. +       slot = slots[i];
  4882. +       slot->rung = rung;
  4883. +       BUG_ON(vma_fully_scanned(slot));
  4884. +   }
  4885. +
  4886. +   if (rung->vma_root.num == num)
  4887. +       reset_current_scan(rung, 0, 1);
  4888. +}
  4889. +
  4890. +static inline int rung_add_one_slot(struct scan_rung *rung,
  4891. +                    struct vma_slot *slot)
  4892. +{
  4893. +   int err;
  4894. +
  4895. +   err = sradix_tree_enter(&rung->vma_root, (void **)&slot, 1);
  4896. +   if (err)
  4897. +       return err;
  4898. +
  4899. +   slot->rung = rung;
  4900. +   if (rung->vma_root.num == 1)
  4901. +       reset_current_scan(rung, 0, 1);
  4902. +
  4903. +   return 0;
  4904. +}
  4905. +
  4906. +/*
  4907. + * Return true if the slot is deleted from its rung.
  4908. + */
  4909. +static inline int vma_rung_enter(struct vma_slot *slot, struct scan_rung *rung)
  4910. +{
  4911. +   struct scan_rung *old_rung = slot->rung;
  4912. +   int err;
  4913. +
  4914. +   if (old_rung == rung)
  4915. +       return 0;
  4916. +
  4917. +   rung_rm_slot(slot);
  4918. +   err = rung_add_one_slot(rung, slot);
  4919. +   if (err) {
  4920. +       err = rung_add_one_slot(old_rung, slot);
  4921. +       WARN_ON(err); /* OOPS, badly OOM, we lost this slot */
  4922. +   }
  4923. +
  4924. +   return 1;
  4925. +}
  4926. +
  4927. +static inline int vma_rung_up(struct vma_slot *slot)
  4928. +{
  4929. +   struct scan_rung *rung;
  4930. +
  4931. +   rung = slot->rung;
  4932. +   if (slot->rung != &uksm_scan_ladder[SCAN_LADDER_SIZE-1])
  4933. +       rung++;
  4934. +
  4935. +   return vma_rung_enter(slot, rung);
  4936. +}
  4937. +
  4938. +static inline int vma_rung_down(struct vma_slot *slot)
  4939. +{
  4940. +   struct scan_rung *rung;
  4941. +
  4942. +   rung = slot->rung;
  4943. +   if (slot->rung != &uksm_scan_ladder[0])
  4944. +       rung--;
  4945. +
  4946. +   return vma_rung_enter(slot, rung);
  4947. +}
  4948. +
  4949. +/**
  4950. + * cal_dedup_ratio() - Calculate the deduplication ratio for this slot.
  4951. + */
  4952. +static unsigned long cal_dedup_ratio(struct vma_slot *slot)
  4953. +{
  4954. +   unsigned long ret;
  4955. +
  4956. +   BUG_ON(slot->pages_scanned == slot->last_scanned);
  4957. +
  4958. +   ret = slot->pages_merged;
  4959. +
  4960. +   /* Thrashing area filtering */
  4961. +   if (ret && uksm_thrash_threshold) {
  4962. +       if (slot->pages_cowed * 100 / slot->pages_merged
  4963. +           > uksm_thrash_threshold) {
  4964. +           ret = 0;
  4965. +       } else {
  4966. +           ret = slot->pages_merged - slot->pages_cowed;
  4967. +       }
  4968. +   }
  4969. +
  4970. +   return ret;
  4971. +}
  4972. +
  4973. +/**
  4974. + * cal_dedup_ratio() - Calculate the deduplication ratio for this slot.
  4975. + */
  4976. +static unsigned long cal_dedup_ratio_old(struct vma_slot *slot)
  4977. +{
  4978. +   unsigned long ret;
  4979. +   unsigned long pages_scanned;
  4980. +
  4981. +   pages_scanned = slot->pages_scanned;
  4982. +   if (!pages_scanned) {
  4983. +       if (uksm_thrash_threshold)
  4984. +           return 0;
  4985. +       else
  4986. +           pages_scanned = slot->pages_scanned;
  4987. +   }
  4988. +
  4989. +   ret = slot->pages_bemerged * 100 / pages_scanned;
  4990. +
  4991. +   /* Thrashing area filtering */
  4992. +   if (ret && uksm_thrash_threshold) {
  4993. +       if (slot->pages_cowed * 100 / slot->pages_bemerged
  4994. +           > uksm_thrash_threshold) {
  4995. +           ret = 0;
  4996. +       } else {
  4997. +           ret = slot->pages_bemerged - slot->pages_cowed;
  4998. +       }
  4999. +   }
  5000. +
  5001. +   return ret;
  5002. +}
  5003. +
  5004. +/**
  5005. + * stable_node_reinsert() - When the hash_strength has been adjusted, the
  5006. + * stable tree need to be restructured, this is the function re-inserting the
  5007. + * stable node.
  5008. + */
  5009. +static inline void stable_node_reinsert(struct stable_node *new_node,
  5010. +                   struct page *page,
  5011. +                   struct rb_root *root_treep,
  5012. +                   struct list_head *tree_node_listp,
  5013. +                   u32 hash)
  5014. +{
  5015. +   struct rb_node **new = &root_treep->rb_node;
  5016. +   struct rb_node *parent = NULL;
  5017. +   struct stable_node *stable_node;
  5018. +   struct tree_node *tree_node;
  5019. +   struct page *tree_page;
  5020. +   int cmp;
  5021. +
  5022. +   while (*new) {
  5023. +       int cmp;
  5024. +
  5025. +       tree_node = rb_entry(*new, struct tree_node, node);
  5026. +
  5027. +       cmp = hash_cmp(hash, tree_node->hash);
  5028. +
  5029. +       if (cmp < 0) {
  5030. +           parent = *new;
  5031. +           new = &parent->rb_left;
  5032. +       } else if (cmp > 0) {
  5033. +           parent = *new;
  5034. +           new = &parent->rb_right;
  5035. +       } else
  5036. +           break;
  5037. +   }
  5038. +
  5039. +   if (*new) {
  5040. +       /* find a stable tree node with same first level hash value */
  5041. +       stable_node_hash_max(new_node, page, hash);
  5042. +       if (tree_node->count == 1) {
  5043. +           stable_node = rb_entry(tree_node->sub_root.rb_node,
  5044. +                          struct stable_node, node);
  5045. +           tree_page = get_uksm_page(stable_node, 1, 0);
  5046. +           if (tree_page) {
  5047. +               stable_node_hash_max(stable_node,
  5048. +                             tree_page, hash);
  5049. +               put_page(tree_page);
  5050. +
  5051. +               /* prepare for stable node insertion */
  5052. +
  5053. +               cmp = hash_cmp(new_node->hash_max,
  5054. +                          stable_node->hash_max);
  5055. +               parent = &stable_node->node;
  5056. +               if (cmp < 0)
  5057. +                   new = &parent->rb_left;
  5058. +               else if (cmp > 0)
  5059. +                   new = &parent->rb_right;
  5060. +               else
  5061. +                   goto failed;
  5062. +
  5063. +               goto add_node;
  5064. +           } else {
  5065. +               /* the only stable_node deleted, the tree node
  5066. +                * was not deleted.
  5067. +                */
  5068. +               goto tree_node_reuse;
  5069. +           }
  5070. +       }
  5071. +
  5072. +       /* well, search the collision subtree */
  5073. +       new = &tree_node->sub_root.rb_node;
  5074. +       parent = NULL;
  5075. +       BUG_ON(!*new);
  5076. +       while (*new) {
  5077. +           int cmp;
  5078. +
  5079. +           stable_node = rb_entry(*new, struct stable_node, node);
  5080. +
  5081. +           cmp = hash_cmp(new_node->hash_max,
  5082. +                      stable_node->hash_max);
  5083. +
  5084. +           if (cmp < 0) {
  5085. +               parent = *new;
  5086. +               new = &parent->rb_left;
  5087. +           } else if (cmp > 0) {
  5088. +               parent = *new;
  5089. +               new = &parent->rb_right;
  5090. +           } else {
  5091. +               /* oh, no, still a collision */
  5092. +               goto failed;
  5093. +           }
  5094. +       }
  5095. +
  5096. +       goto add_node;
  5097. +   }
  5098. +
  5099. +   /* no tree node found */
  5100. +   tree_node = alloc_tree_node(tree_node_listp);
  5101. +   if (!tree_node) {
  5102. +       printk(KERN_ERR "UKSM: memory allocation error!\n");
  5103. +       goto failed;
  5104. +   } else {
  5105. +       tree_node->hash = hash;
  5106. +       rb_link_node(&tree_node->node, parent, new);
  5107. +       rb_insert_color(&tree_node->node, root_treep);
  5108. +
  5109. +tree_node_reuse:
  5110. +       /* prepare for stable node insertion */
  5111. +       parent = NULL;
  5112. +       new = &tree_node->sub_root.rb_node;
  5113. +   }
  5114. +
  5115. +add_node:
  5116. +   rb_link_node(&new_node->node, parent, new);
  5117. +   rb_insert_color(&new_node->node, &tree_node->sub_root);
  5118. +   new_node->tree_node = tree_node;
  5119. +   tree_node->count++;
  5120. +   return;
  5121. +
  5122. +failed:
  5123. +   /* This can only happen when two nodes have collided
  5124. +    * in two levels.
  5125. +    */
  5126. +   new_node->tree_node = NULL;
  5127. +   return;
  5128. +}
  5129. +
  5130. +static inline void free_all_tree_nodes(struct list_head *list)
  5131. +{
  5132. +   struct tree_node *node, *tmp;
  5133. +
  5134. +   list_for_each_entry_safe(node, tmp, list, all_list) {
  5135. +       free_tree_node(node);
  5136. +   }
  5137. +}
  5138. +
  5139. +/**
  5140. + * stable_tree_delta_hash() - Delta hash the stable tree from previous hash
  5141. + * strength to the current hash_strength. It re-structures the hole tree.
  5142. + */
  5143. +static inline void stable_tree_delta_hash(u32 prev_hash_strength)
  5144. +{
  5145. +   struct stable_node *node, *tmp;
  5146. +   struct rb_root *root_new_treep;
  5147. +   struct list_head *new_tree_node_listp;
  5148. +
  5149. +   stable_tree_index = (stable_tree_index + 1) % 2;
  5150. +   root_new_treep = &root_stable_tree[stable_tree_index];
  5151. +   new_tree_node_listp = &stable_tree_node_list[stable_tree_index];
  5152. +   *root_new_treep = RB_ROOT;
  5153. +   BUG_ON(!list_empty(new_tree_node_listp));
  5154. +
  5155. +   /*
  5156. +    * we need to be safe, the node could be removed by get_uksm_page()
  5157. +    */
  5158. +   list_for_each_entry_safe(node, tmp, &stable_node_list, all_list) {
  5159. +       void *addr;
  5160. +       struct page *node_page;
  5161. +       u32 hash;
  5162. +
  5163. +       /*
  5164. +        * We are completely re-structuring the stable nodes to a new
  5165. +        * stable tree. We don't want to touch the old tree unlinks and
  5166. +        * old tree_nodes. The old tree_nodes will be freed at once.
  5167. +        */
  5168. +       node_page = get_uksm_page(node, 0, 0);
  5169. +       if (!node_page)
  5170. +           continue;
  5171. +
  5172. +       if (node->tree_node) {
  5173. +           hash = node->tree_node->hash;
  5174. +
  5175. +           addr = kmap_atomic(node_page);
  5176. +
  5177. +           hash = delta_hash(addr, prev_hash_strength,
  5178. +                     hash_strength, hash);
  5179. +           kunmap_atomic(addr);
  5180. +       } else {
  5181. +           /*
  5182. +            *it was not inserted to rbtree due to collision in last
  5183. +            *round scan.
  5184. +            */
  5185. +           hash = page_hash(node_page, hash_strength, 0);
  5186. +       }
  5187. +
  5188. +       stable_node_reinsert(node, node_page, root_new_treep,
  5189. +                    new_tree_node_listp, hash);
  5190. +       put_page(node_page);
  5191. +   }
  5192. +
  5193. +   root_stable_treep = root_new_treep;
  5194. +   free_all_tree_nodes(stable_tree_node_listp);
  5195. +   BUG_ON(!list_empty(stable_tree_node_listp));
  5196. +   stable_tree_node_listp = new_tree_node_listp;
  5197. +}
  5198. +
  5199. +static inline void inc_hash_strength(unsigned long delta)
  5200. +{
  5201. +   hash_strength += 1 << delta;
  5202. +   if (hash_strength > HASH_STRENGTH_MAX)
  5203. +       hash_strength = HASH_STRENGTH_MAX;
  5204. +}
  5205. +
  5206. +static inline void dec_hash_strength(unsigned long delta)
  5207. +{
  5208. +   unsigned long change = 1 << delta;
  5209. +
  5210. +   if (hash_strength <= change + 1)
  5211. +       hash_strength = 1;
  5212. +   else
  5213. +       hash_strength -= change;
  5214. +}
  5215. +
  5216. +static inline void inc_hash_strength_delta(void)
  5217. +{
  5218. +   hash_strength_delta++;
  5219. +   if (hash_strength_delta > HASH_STRENGTH_DELTA_MAX)
  5220. +       hash_strength_delta = HASH_STRENGTH_DELTA_MAX;
  5221. +}
  5222. +
  5223. +/*
  5224. +static inline unsigned long get_current_neg_ratio(void)
  5225. +{
  5226. +   if (!rshash_pos || rshash_neg > rshash_pos)
  5227. +       return 100;
  5228. +
  5229. +   return div64_u64(100 * rshash_neg , rshash_pos);
  5230. +}
  5231. +*/
  5232. +
  5233. +static inline unsigned long get_current_neg_ratio(void)
  5234. +{
  5235. +   u64 pos = benefit.pos;
  5236. +   u64 neg = benefit.neg;
  5237. +
  5238. +   if (!neg)
  5239. +       return 0;
  5240. +
  5241. +   if (!pos || neg > pos)
  5242. +       return 100;
  5243. +
  5244. +   if (neg > div64_u64(U64_MAX, 100))
  5245. +       pos = div64_u64(pos, 100);
  5246. +   else
  5247. +       neg *= 100;
  5248. +
  5249. +   return div64_u64(neg, pos);
  5250. +}
  5251. +
  5252. +static inline unsigned long get_current_benefit(void)
  5253. +{
  5254. +   u64 pos = benefit.pos;
  5255. +   u64 neg = benefit.neg;
  5256. +   u64 scanned = benefit.scanned;
  5257. +
  5258. +   if (neg > pos)
  5259. +       return 0;
  5260. +
  5261. +   return div64_u64((pos - neg), scanned);
  5262. +}
  5263. +
  5264. +static inline int judge_rshash_direction(void)
  5265. +{
  5266. +   u64 current_neg_ratio, stable_benefit;
  5267. +   u64 current_benefit, delta = 0;
  5268. +   int ret = STILL;
  5269. +
  5270. +   /* Try to probe a value after the boot, and in case the system
  5271. +      are still for a long time. */
  5272. +   if ((fully_scanned_round & 0xFFULL) == 10) {
  5273. +       ret = OBSCURE;
  5274. +       goto out;
  5275. +   }
  5276. +
  5277. +   current_neg_ratio = get_current_neg_ratio();
  5278. +
  5279. +   if (current_neg_ratio == 0) {
  5280. +       rshash_neg_cont_zero++;
  5281. +       if (rshash_neg_cont_zero > 2)
  5282. +           return GO_DOWN;
  5283. +       else
  5284. +           return STILL;
  5285. +   }
  5286. +   rshash_neg_cont_zero = 0;
  5287. +
  5288. +   if (current_neg_ratio > 90) {
  5289. +       ret = GO_UP;
  5290. +       goto out;
  5291. +   }
  5292. +
  5293. +   current_benefit = get_current_benefit();
  5294. +   stable_benefit = rshash_state.stable_benefit;
  5295. +
  5296. +   if (!stable_benefit) {
  5297. +       ret = OBSCURE;
  5298. +       goto out;
  5299. +   }
  5300. +
  5301. +   if (current_benefit > stable_benefit)
  5302. +       delta = current_benefit - stable_benefit;
  5303. +   else if (current_benefit < stable_benefit)
  5304. +       delta = stable_benefit - current_benefit;
  5305. +
  5306. +   delta = div64_u64(100 * delta , stable_benefit);
  5307. +
  5308. +   if (delta > 50) {
  5309. +       rshash_cont_obscure++;
  5310. +       if (rshash_cont_obscure > 2)
  5311. +           return OBSCURE;
  5312. +       else
  5313. +           return STILL;
  5314. +   }
  5315. +
  5316. +out:
  5317. +   rshash_cont_obscure = 0;
  5318. +   return ret;
  5319. +}
  5320. +
  5321. +/**
  5322. + * rshash_adjust() - The main function to control the random sampling state
  5323. + * machine for hash strength adapting.
  5324. + *
  5325. + * return true if hash_strength has changed.
  5326. + */
  5327. +static inline int rshash_adjust(void)
  5328. +{
  5329. +   unsigned long prev_hash_strength = hash_strength;
  5330. +
  5331. +   if (!encode_benefit())
  5332. +       return 0;
  5333. +
  5334. +   switch (rshash_state.state) {
  5335. +   case RSHASH_STILL:
  5336. +       switch (judge_rshash_direction()) {
  5337. +       case GO_UP:
  5338. +           if (rshash_state.pre_direct == GO_DOWN)
  5339. +               hash_strength_delta = 0;
  5340. +
  5341. +           inc_hash_strength(hash_strength_delta);
  5342. +           inc_hash_strength_delta();
  5343. +           rshash_state.stable_benefit = get_current_benefit();
  5344. +           rshash_state.pre_direct = GO_UP;
  5345. +           break;
  5346. +
  5347. +       case GO_DOWN:
  5348. +           if (rshash_state.pre_direct == GO_UP)
  5349. +               hash_strength_delta = 0;
  5350. +
  5351. +           dec_hash_strength(hash_strength_delta);
  5352. +           inc_hash_strength_delta();
  5353. +           rshash_state.stable_benefit = get_current_benefit();
  5354. +           rshash_state.pre_direct = GO_DOWN;
  5355. +           break;
  5356. +
  5357. +       case OBSCURE:
  5358. +           rshash_state.stable_point = hash_strength;
  5359. +           rshash_state.turn_point_down = hash_strength;
  5360. +           rshash_state.turn_point_up = hash_strength;
  5361. +           rshash_state.turn_benefit_down = get_current_benefit();
  5362. +           rshash_state.turn_benefit_up = get_current_benefit();
  5363. +           rshash_state.lookup_window_index = 0;
  5364. +           rshash_state.state = RSHASH_TRYDOWN;
  5365. +           dec_hash_strength(hash_strength_delta);
  5366. +           inc_hash_strength_delta();
  5367. +           break;
  5368. +
  5369. +       case STILL:
  5370. +           break;
  5371. +       default:
  5372. +           BUG();
  5373. +       }
  5374. +       break;
  5375. +
  5376. +   case RSHASH_TRYDOWN:
  5377. +       if (rshash_state.lookup_window_index++ % 5 == 0)
  5378. +           rshash_state.below_count = 0;
  5379. +
  5380. +       if (get_current_benefit() < rshash_state.stable_benefit)
  5381. +           rshash_state.below_count++;
  5382. +       else if (get_current_benefit() >
  5383. +            rshash_state.turn_benefit_down) {
  5384. +           rshash_state.turn_point_down = hash_strength;
  5385. +           rshash_state.turn_benefit_down = get_current_benefit();
  5386. +       }
  5387. +
  5388. +       if (rshash_state.below_count >= 3 ||
  5389. +           judge_rshash_direction() == GO_UP ||
  5390. +           hash_strength == 1) {
  5391. +           hash_strength = rshash_state.stable_point;
  5392. +           hash_strength_delta = 0;
  5393. +           inc_hash_strength(hash_strength_delta);
  5394. +           inc_hash_strength_delta();
  5395. +           rshash_state.lookup_window_index = 0;
  5396. +           rshash_state.state = RSHASH_TRYUP;
  5397. +           hash_strength_delta = 0;
  5398. +       } else {
  5399. +           dec_hash_strength(hash_strength_delta);
  5400. +           inc_hash_strength_delta();
  5401. +       }
  5402. +       break;
  5403. +
  5404. +   case RSHASH_TRYUP:
  5405. +       if (rshash_state.lookup_window_index++ % 5 == 0)
  5406. +           rshash_state.below_count = 0;
  5407. +
  5408. +       if (get_current_benefit() < rshash_state.turn_benefit_down)
  5409. +           rshash_state.below_count++;
  5410. +       else if (get_current_benefit() > rshash_state.turn_benefit_up) {
  5411. +           rshash_state.turn_point_up = hash_strength;
  5412. +           rshash_state.turn_benefit_up = get_current_benefit();
  5413. +       }
  5414. +
  5415. +       if (rshash_state.below_count >= 3 ||
  5416. +           judge_rshash_direction() == GO_DOWN ||
  5417. +           hash_strength == HASH_STRENGTH_MAX) {
  5418. +           hash_strength = rshash_state.turn_benefit_up >
  5419. +               rshash_state.turn_benefit_down ?
  5420. +               rshash_state.turn_point_up :
  5421. +               rshash_state.turn_point_down;
  5422. +
  5423. +           rshash_state.state = RSHASH_PRE_STILL;
  5424. +       } else {
  5425. +           inc_hash_strength(hash_strength_delta);
  5426. +           inc_hash_strength_delta();
  5427. +       }
  5428. +
  5429. +       break;
  5430. +
  5431. +   case RSHASH_NEW:
  5432. +   case RSHASH_PRE_STILL:
  5433. +       rshash_state.stable_benefit = get_current_benefit();
  5434. +       rshash_state.state = RSHASH_STILL;
  5435. +       hash_strength_delta = 0;
  5436. +       break;
  5437. +   default:
  5438. +       BUG();
  5439. +   }
  5440. +
  5441. +   /* rshash_neg = rshash_pos = 0; */
  5442. +   reset_benefit();
  5443. +
  5444. +   if (prev_hash_strength != hash_strength)
  5445. +       stable_tree_delta_hash(prev_hash_strength);
  5446. +
  5447. +   return prev_hash_strength != hash_strength;
  5448. +}
  5449. +
  5450. +/**
  5451. + * round_update_ladder() - The main function to do update of all the
  5452. + * adjustments whenever a scan round is finished.
  5453. + */
  5454. +static noinline void round_update_ladder(void)
  5455. +{
  5456. +   int i;
  5457. +   unsigned long dedup;
  5458. +   struct vma_slot *slot, *tmp_slot;
  5459. +
  5460. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  5461. +       uksm_scan_ladder[i].flags &= ~UKSM_RUNG_ROUND_FINISHED;
  5462. +   }
  5463. +
  5464. +   list_for_each_entry_safe(slot, tmp_slot, &vma_slot_dedup, dedup_list) {
  5465. +
  5466. +       /* slot may be rung_rm_slot() when mm exits */
  5467. +       if (slot->snode) {
  5468. +           dedup = cal_dedup_ratio_old(slot);
  5469. +           if (dedup && dedup >= uksm_abundant_threshold)
  5470. +               vma_rung_up(slot);
  5471. +       }
  5472. +
  5473. +       slot->pages_bemerged = 0;
  5474. +       slot->pages_cowed = 0;
  5475. +
  5476. +       list_del_init(&slot->dedup_list);
  5477. +   }
  5478. +}
  5479. +
  5480. +static void uksm_del_vma_slot(struct vma_slot *slot)
  5481. +{
  5482. +   int i, j;
  5483. +   struct rmap_list_entry *entry;
  5484. +
  5485. +   if (slot->snode) {
  5486. +       /*
  5487. +        * In case it just failed when entering the rung, it's not
  5488. +        * necessary.
  5489. +        */
  5490. +       rung_rm_slot(slot);
  5491. +   }
  5492. +
  5493. +   if (!list_empty(&slot->dedup_list))
  5494. +       list_del(&slot->dedup_list);
  5495. +
  5496. +   if (!slot->rmap_list_pool || !slot->pool_counts) {
  5497. +       /* In case it OOMed in uksm_vma_enter() */
  5498. +       goto out;
  5499. +   }
  5500. +
  5501. +   for (i = 0; i < slot->pool_size; i++) {
  5502. +       void *addr;
  5503. +
  5504. +       if (!slot->rmap_list_pool[i])
  5505. +           continue;
  5506. +
  5507. +       addr = kmap(slot->rmap_list_pool[i]);
  5508. +       for (j = 0; j < PAGE_SIZE / sizeof(*entry); j++) {
  5509. +           entry = (struct rmap_list_entry *)addr + j;
  5510. +           if (is_addr(entry->addr))
  5511. +               continue;
  5512. +           if (!entry->item)
  5513. +               continue;
  5514. +
  5515. +           remove_rmap_item_from_tree(entry->item);
  5516. +           free_rmap_item(entry->item);
  5517. +           slot->pool_counts[i]--;
  5518. +       }
  5519. +       BUG_ON(slot->pool_counts[i]);
  5520. +       kunmap(slot->rmap_list_pool[i]);
  5521. +       __free_page(slot->rmap_list_pool[i]);
  5522. +   }
  5523. +   kfree(slot->rmap_list_pool);
  5524. +   kfree(slot->pool_counts);
  5525. +
  5526. +out:
  5527. +   slot->rung = NULL;
  5528. +   BUG_ON(uksm_pages_total < slot->pages);
  5529. +   if (slot->flags & UKSM_SLOT_IN_UKSM)
  5530. +       uksm_pages_total -= slot->pages;
  5531. +
  5532. +   if (slot->fully_scanned_round == fully_scanned_round)
  5533. +       scanned_virtual_pages -= slot->pages;
  5534. +   else
  5535. +       scanned_virtual_pages -= slot->pages_scanned;
  5536. +   free_vma_slot(slot);
  5537. +}
  5538. +
  5539. +
  5540. +#define SPIN_LOCK_PERIOD   32
  5541. +static struct vma_slot *cleanup_slots[SPIN_LOCK_PERIOD];
  5542. +static inline void cleanup_vma_slots(void)
  5543. +{
  5544. +   struct vma_slot *slot;
  5545. +   int i;
  5546. +
  5547. +   i = 0;
  5548. +   spin_lock(&vma_slot_list_lock);
  5549. +   while (!list_empty(&vma_slot_del)) {
  5550. +       slot = list_entry(vma_slot_del.next,
  5551. +                 struct vma_slot, slot_list);
  5552. +       list_del(&slot->slot_list);
  5553. +       cleanup_slots[i++] = slot;
  5554. +       if (i == SPIN_LOCK_PERIOD) {
  5555. +           spin_unlock(&vma_slot_list_lock);
  5556. +           while (--i >= 0)
  5557. +               uksm_del_vma_slot(cleanup_slots[i]);
  5558. +           i = 0;
  5559. +           spin_lock(&vma_slot_list_lock);
  5560. +       }
  5561. +   }
  5562. +   spin_unlock(&vma_slot_list_lock);
  5563. +
  5564. +   while (--i >= 0)
  5565. +       uksm_del_vma_slot(cleanup_slots[i]);
  5566. +}
  5567. +
  5568. +/*
  5569. +*expotional moving average formula
  5570. +*/
  5571. +static inline unsigned long ema(unsigned long curr, unsigned long last_ema)
  5572. +{
  5573. +   /*
  5574. +    * For a very high burst, even the ema cannot work well, a false very
  5575. +    * high per-page time estimation can result in feedback in very high
  5576. +    * overhead of context swith and rung update -- this will then lead
  5577. +    * to higher per-paper time, this may not converge.
  5578. +    *
  5579. +    * Instead, we try to approach this value in a binary manner.
  5580. +    */
  5581. +   if (curr > last_ema * 10)
  5582. +       return last_ema * 2;
  5583. +
  5584. +   return (EMA_ALPHA * curr + (100 - EMA_ALPHA) * last_ema) / 100;
  5585. +}
  5586. +
  5587. +/*
  5588. + * convert cpu ratio in 1/TIME_RATIO_SCALE configured by user to
  5589. + * nanoseconds based on current uksm_sleep_jiffies.
  5590. + */
  5591. +static inline unsigned long cpu_ratio_to_nsec(unsigned int ratio)
  5592. +{
  5593. +   return NSEC_PER_USEC * jiffies_to_usecs(uksm_sleep_jiffies) /
  5594. +       (TIME_RATIO_SCALE - ratio) * ratio;
  5595. +}
  5596. +
  5597. +
  5598. +static inline unsigned long rung_real_ratio(int cpu_time_ratio)
  5599. +{
  5600. +   unsigned long ret;
  5601. +
  5602. +   BUG_ON(!cpu_time_ratio);
  5603. +
  5604. +   if (cpu_time_ratio > 0)
  5605. +       ret = cpu_time_ratio;
  5606. +   else
  5607. +       ret = (unsigned long)(-cpu_time_ratio) *
  5608. +           uksm_max_cpu_percentage / 100UL;
  5609. +
  5610. +   return ret ? ret : 1;
  5611. +}
  5612. +
  5613. +static noinline void uksm_calc_scan_pages(void)
  5614. +{
  5615. +   struct scan_rung *ladder = uksm_scan_ladder;
  5616. +   unsigned long sleep_usecs, nsecs;
  5617. +   unsigned long ratio;
  5618. +   int i;
  5619. +   unsigned long per_page;
  5620. +
  5621. +   if (uksm_ema_page_time > 100000 ||
  5622. +       (((unsigned long) uksm_eval_round & (256UL - 1)) == 0UL))
  5623. +       uksm_ema_page_time = UKSM_PAGE_TIME_DEFAULT;
  5624. +
  5625. +   per_page = uksm_ema_page_time;
  5626. +   BUG_ON(!per_page);
  5627. +
  5628. +   /*
  5629. +    * For every 8 eval round, we try to probe a uksm_sleep_jiffies value
  5630. +    * based on saved user input.
  5631. +    */
  5632. +   if (((unsigned long) uksm_eval_round & (8UL - 1)) == 0UL)
  5633. +       uksm_sleep_jiffies = uksm_sleep_saved;
  5634. +
  5635. +   /* We require a rung scan at least 1 page in a period. */
  5636. +   nsecs = per_page;
  5637. +   ratio = rung_real_ratio(ladder[0].cpu_ratio);
  5638. +   if (cpu_ratio_to_nsec(ratio) < nsecs) {
  5639. +       sleep_usecs = nsecs * (TIME_RATIO_SCALE - ratio) / ratio
  5640. +               / NSEC_PER_USEC;
  5641. +       uksm_sleep_jiffies = usecs_to_jiffies(sleep_usecs) + 1;
  5642. +   }
  5643. +
  5644. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  5645. +       ratio = rung_real_ratio(ladder[i].cpu_ratio);
  5646. +       ladder[i].pages_to_scan = cpu_ratio_to_nsec(ratio) /
  5647. +                   per_page;
  5648. +       BUG_ON(!ladder[i].pages_to_scan);
  5649. +       uksm_calc_rung_step(&ladder[i], per_page, ratio);
  5650. +   }
  5651. +}
  5652. +
  5653. +/*
  5654. + * From the scan time of this round (ns) to next expected min sleep time
  5655. + * (ms), be careful of the possible overflows. ratio is taken from
  5656. + * rung_real_ratio()
  5657. + */
  5658. +static inline
  5659. +unsigned int scan_time_to_sleep(unsigned long long scan_time, unsigned long ratio)
  5660. +{
  5661. +   scan_time >>= 20; /* to msec level now */
  5662. +   BUG_ON(scan_time > (ULONG_MAX / TIME_RATIO_SCALE));
  5663. +
  5664. +   return (unsigned int) ((unsigned long) scan_time *
  5665. +                  (TIME_RATIO_SCALE - ratio) / ratio);
  5666. +}
  5667. +
  5668. +#define __round_mask(x, y) ((__typeof__(x))((y)-1))
  5669. +#define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1)
  5670. +
  5671. +static inline unsigned long vma_pool_size(struct vma_slot *slot)
  5672. +{
  5673. +   return round_up(sizeof(struct rmap_list_entry) * slot->pages,
  5674. +           PAGE_SIZE) >> PAGE_SHIFT;
  5675. +}
  5676. +
  5677. +static void uksm_vma_enter(struct vma_slot **slots, unsigned long num)
  5678. +{
  5679. +   struct scan_rung *rung;
  5680. +   unsigned long pool_size, i;
  5681. +   struct vma_slot *slot;
  5682. +   int failed;
  5683. +
  5684. +   rung = &uksm_scan_ladder[0];
  5685. +
  5686. +   failed = 0;
  5687. +   for (i = 0; i < num; i++) {
  5688. +       slot = slots[i];
  5689. +
  5690. +       pool_size = vma_pool_size(slot);
  5691. +       slot->rmap_list_pool = kzalloc(sizeof(struct page *) *
  5692. +                          pool_size, GFP_KERNEL);
  5693. +       if (!slot->rmap_list_pool)
  5694. +           break;
  5695. +
  5696. +       slot->pool_counts = kzalloc(sizeof(unsigned int) * pool_size,
  5697. +                       GFP_KERNEL);
  5698. +       if (!slot->pool_counts) {
  5699. +           kfree(slot->rmap_list_pool);
  5700. +           break;
  5701. +       }
  5702. +
  5703. +       slot->pool_size = pool_size;
  5704. +       BUG_ON(CAN_OVERFLOW_U64(uksm_pages_total, slot->pages));
  5705. +       slot->flags |= UKSM_SLOT_IN_UKSM;
  5706. +       uksm_pages_total += slot->pages;
  5707. +   }
  5708. +
  5709. +   if (i)
  5710. +       rung_add_new_slots(rung, slots, i);
  5711. +
  5712. +   return;
  5713. +}
  5714. +
  5715. +static struct vma_slot *batch_slots[SLOT_TREE_NODE_STORE_SIZE];
  5716. +
  5717. +static void uksm_enter_all_slots(void)
  5718. +{
  5719. +   struct vma_slot *slot;
  5720. +   unsigned long index;
  5721. +   struct list_head empty_vma_list;
  5722. +   int i;
  5723. +
  5724. +   i = 0;
  5725. +   index = 0;
  5726. +   INIT_LIST_HEAD(&empty_vma_list);
  5727. +
  5728. +   spin_lock(&vma_slot_list_lock);
  5729. +   while (!list_empty(&vma_slot_new)) {
  5730. +       slot = list_entry(vma_slot_new.next,
  5731. +                 struct vma_slot, slot_list);
  5732. +
  5733. +       if (!slot->vma->anon_vma) {
  5734. +           list_move(&slot->slot_list, &empty_vma_list);
  5735. +       } else if (vma_can_enter(slot->vma)) {
  5736. +           batch_slots[index++] = slot;
  5737. +           list_del_init(&slot->slot_list);
  5738. +       } else {
  5739. +           list_move(&slot->slot_list, &vma_slot_noadd);
  5740. +       }
  5741. +
  5742. +       if (++i == SPIN_LOCK_PERIOD ||
  5743. +           (index && !(index % SLOT_TREE_NODE_STORE_SIZE))) {
  5744. +           spin_unlock(&vma_slot_list_lock);
  5745. +
  5746. +           if (index && !(index % SLOT_TREE_NODE_STORE_SIZE)) {
  5747. +               uksm_vma_enter(batch_slots, index);
  5748. +               index = 0;
  5749. +           }
  5750. +           i = 0;
  5751. +           cond_resched();
  5752. +           spin_lock(&vma_slot_list_lock);
  5753. +       }
  5754. +   }
  5755. +
  5756. +   list_splice(&empty_vma_list, &vma_slot_new);
  5757. +
  5758. +   spin_unlock(&vma_slot_list_lock);
  5759. +
  5760. +   if (index)
  5761. +       uksm_vma_enter(batch_slots, index);
  5762. +
  5763. +}
  5764. +
  5765. +static inline int rung_round_finished(struct scan_rung *rung)
  5766. +{
  5767. +   return rung->flags & UKSM_RUNG_ROUND_FINISHED;
  5768. +}
  5769. +
  5770. +static inline void judge_slot(struct vma_slot *slot)
  5771. +{
  5772. +   struct scan_rung *rung = slot->rung;
  5773. +   unsigned long dedup;
  5774. +   int deleted;
  5775. +
  5776. +   dedup = cal_dedup_ratio(slot);
  5777. +   if (vma_fully_scanned(slot) && uksm_thrash_threshold)
  5778. +       deleted = vma_rung_enter(slot, &uksm_scan_ladder[0]);
  5779. +   else if (dedup && dedup >= uksm_abundant_threshold)
  5780. +       deleted = vma_rung_up(slot);
  5781. +   else
  5782. +       deleted = vma_rung_down(slot);
  5783. +
  5784. +   slot->pages_merged = 0;
  5785. +   slot->pages_cowed = 0;
  5786. +
  5787. +   if (vma_fully_scanned(slot))
  5788. +       slot->pages_scanned = 0;
  5789. +
  5790. +   slot->last_scanned = slot->pages_scanned;
  5791. +
  5792. +   /* If its deleted in above, then rung was already advanced. */
  5793. +   if (!deleted)
  5794. +       advance_current_scan(rung);
  5795. +}
  5796. +
  5797. +
  5798. +static inline int hash_round_finished(void)
  5799. +{
  5800. +   if (scanned_virtual_pages > (uksm_pages_total >> 2)) {
  5801. +       scanned_virtual_pages = 0;
  5802. +       if (uksm_pages_scanned)
  5803. +           fully_scanned_round++;
  5804. +
  5805. +       return 1;
  5806. +   } else {
  5807. +       return 0;
  5808. +   }
  5809. +}
  5810. +
  5811. +#define UKSM_MMSEM_BATCH   5
  5812. +#define BUSY_RETRY     100
  5813. +
  5814. +/**
  5815. + * uksm_do_scan()  - the main worker function.
  5816. + */
  5817. +static noinline void uksm_do_scan(void)
  5818. +{
  5819. +   struct vma_slot *slot, *iter;
  5820. +   struct mm_struct *busy_mm;
  5821. +   unsigned char round_finished, all_rungs_emtpy;
  5822. +   int i, err, mmsem_batch;
  5823. +   unsigned long pcost;
  5824. +   long long delta_exec;
  5825. +   unsigned long vpages, max_cpu_ratio;
  5826. +   unsigned long long start_time, end_time, scan_time;
  5827. +   unsigned int expected_jiffies;
  5828. +
  5829. +   might_sleep();
  5830. +
  5831. +   vpages = 0;
  5832. +
  5833. +   start_time = task_sched_runtime(current);
  5834. +   max_cpu_ratio = 0;
  5835. +   mmsem_batch = 0;
  5836. +
  5837. +   for (i = 0; i < SCAN_LADDER_SIZE;) {
  5838. +       struct scan_rung *rung = &uksm_scan_ladder[i];
  5839. +       unsigned long ratio;
  5840. +       int busy_retry;
  5841. +
  5842. +       if (!rung->pages_to_scan) {
  5843. +           i++;
  5844. +           continue;
  5845. +       }
  5846. +
  5847. +       if (!rung->vma_root.num) {
  5848. +           rung->pages_to_scan = 0;
  5849. +           i++;
  5850. +           continue;
  5851. +       }
  5852. +
  5853. +       ratio = rung_real_ratio(rung->cpu_ratio);
  5854. +       if (ratio > max_cpu_ratio)
  5855. +           max_cpu_ratio = ratio;
  5856. +
  5857. +       busy_retry = BUSY_RETRY;
  5858. +       /*
  5859. +        * Do not consider rung_round_finished() here, just used up the
  5860. +        * rung->pages_to_scan quota.
  5861. +        */
  5862. +       while (rung->pages_to_scan && rung->vma_root.num &&
  5863. +              likely(!freezing(current))) {
  5864. +           int reset = 0;
  5865. +
  5866. +           slot = rung->current_scan;
  5867. +
  5868. +           BUG_ON(vma_fully_scanned(slot));
  5869. +
  5870. +           if (mmsem_batch) {
  5871. +               err = 0;
  5872. +           } else {
  5873. +               err = try_down_read_slot_mmap_sem(slot);
  5874. +           }
  5875. +
  5876. +           if (err == -ENOENT) {
  5877. +rm_slot:
  5878. +               rung_rm_slot(slot);
  5879. +               continue;
  5880. +           }
  5881. +
  5882. +           busy_mm = slot->mm;
  5883. +
  5884. +           if (err == -EBUSY) {
  5885. +               /* skip other vmas on the same mm */
  5886. +               do {
  5887. +                   reset = advance_current_scan(rung);
  5888. +                   iter = rung->current_scan;
  5889. +                   busy_retry--;
  5890. +                   if (iter->vma->vm_mm != busy_mm ||
  5891. +                       !busy_retry || reset)
  5892. +                       break;
  5893. +               } while (1);
  5894. +
  5895. +               if (iter->vma->vm_mm != busy_mm) {
  5896. +                   continue;
  5897. +               } else {
  5898. +                   /* scan round finsished */
  5899. +                   break;
  5900. +               }
  5901. +           }
  5902. +
  5903. +           BUG_ON(!vma_can_enter(slot->vma));
  5904. +           if (uksm_test_exit(slot->vma->vm_mm)) {
  5905. +               mmsem_batch = 0;
  5906. +               up_read(&slot->vma->vm_mm->mmap_sem);
  5907. +               goto rm_slot;
  5908. +           }
  5909. +
  5910. +           if (mmsem_batch)
  5911. +               mmsem_batch--;
  5912. +           else
  5913. +               mmsem_batch = UKSM_MMSEM_BATCH;
  5914. +
  5915. +           /* Ok, we have take the mmap_sem, ready to scan */
  5916. +           scan_vma_one_page(slot);
  5917. +           rung->pages_to_scan--;
  5918. +           vpages++;
  5919. +
  5920. +           if (rung->current_offset + rung->step > slot->pages - 1
  5921. +               || vma_fully_scanned(slot)) {
  5922. +               up_read(&slot->vma->vm_mm->mmap_sem);
  5923. +               judge_slot(slot);
  5924. +               mmsem_batch = 0;
  5925. +           } else {
  5926. +               rung->current_offset += rung->step;
  5927. +               if (!mmsem_batch)
  5928. +                   up_read(&slot->vma->vm_mm->mmap_sem);
  5929. +           }
  5930. +
  5931. +           busy_retry = BUSY_RETRY;
  5932. +           cond_resched();
  5933. +       }
  5934. +
  5935. +       if (mmsem_batch) {
  5936. +           up_read(&slot->vma->vm_mm->mmap_sem);
  5937. +           mmsem_batch = 0;
  5938. +       }
  5939. +
  5940. +       if (freezing(current))
  5941. +           break;
  5942. +
  5943. +       cond_resched();
  5944. +   }
  5945. +   end_time = task_sched_runtime(current);
  5946. +   delta_exec = end_time - start_time;
  5947. +
  5948. +   if (freezing(current))
  5949. +       return;
  5950. +
  5951. +   cleanup_vma_slots();
  5952. +   uksm_enter_all_slots();
  5953. +
  5954. +   round_finished = 1;
  5955. +   all_rungs_emtpy = 1;
  5956. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  5957. +       struct scan_rung *rung = &uksm_scan_ladder[i];
  5958. +
  5959. +       if (rung->vma_root.num) {
  5960. +           all_rungs_emtpy = 0;
  5961. +           if (!rung_round_finished(rung))
  5962. +               round_finished = 0;
  5963. +       }
  5964. +   }
  5965. +
  5966. +   if (all_rungs_emtpy)
  5967. +       round_finished = 0;
  5968. +
  5969. +   if (round_finished) {
  5970. +       round_update_ladder();
  5971. +       uksm_eval_round++;
  5972. +
  5973. +       if (hash_round_finished() && rshash_adjust()) {
  5974. +           /* Reset the unstable root iff hash strength changed */
  5975. +           uksm_hash_round++;
  5976. +           root_unstable_tree = RB_ROOT;
  5977. +           free_all_tree_nodes(&unstable_tree_node_list);
  5978. +       }
  5979. +
  5980. +       /*
  5981. +        * A number of pages can hang around indefinitely on per-cpu
  5982. +        * pagevecs, raised page count preventing write_protect_page
  5983. +        * from merging them.  Though it doesn't really matter much,
  5984. +        * it is puzzling to see some stuck in pages_volatile until
  5985. +        * other activity jostles them out, and they also prevented
  5986. +        * LTP's KSM test from succeeding deterministically; so drain
  5987. +        * them here (here rather than on entry to uksm_do_scan(),
  5988. +        * so we don't IPI too often when pages_to_scan is set low).
  5989. +        */
  5990. +       lru_add_drain_all();
  5991. +   }
  5992. +
  5993. +
  5994. +   if (vpages && delta_exec > 0) {
  5995. +       pcost = (unsigned long) delta_exec / vpages;
  5996. +       if (likely(uksm_ema_page_time))
  5997. +           uksm_ema_page_time = ema(pcost, uksm_ema_page_time);
  5998. +       else
  5999. +           uksm_ema_page_time = pcost;
  6000. +   }
  6001. +
  6002. +   uksm_calc_scan_pages();
  6003. +   uksm_sleep_real = uksm_sleep_jiffies;
  6004. +   /* in case of radical cpu bursts, apply the upper bound */
  6005. +   end_time = task_sched_runtime(current);
  6006. +   if (max_cpu_ratio && end_time > start_time) {
  6007. +       scan_time = end_time - start_time;
  6008. +       expected_jiffies = msecs_to_jiffies(
  6009. +           scan_time_to_sleep(scan_time, max_cpu_ratio));
  6010. +
  6011. +       if (expected_jiffies > uksm_sleep_real)
  6012. +           uksm_sleep_real = expected_jiffies;
  6013. +
  6014. +       /* We have a 1 second up bound for responsiveness. */
  6015. +       if (jiffies_to_msecs(uksm_sleep_real) > MSEC_PER_SEC)
  6016. +           uksm_sleep_real = msecs_to_jiffies(1000);
  6017. +   }
  6018. +
  6019. +   return;
  6020. +}
  6021. +
  6022. +static int ksmd_should_run(void)
  6023. +{
  6024. +   return uksm_run & UKSM_RUN_MERGE;
  6025. +}
  6026. +
  6027. +static int uksm_scan_thread(void *nothing)
  6028. +{
  6029. +   set_freezable();
  6030. +   set_user_nice(current, 5);
  6031. +
  6032. +   while (!kthread_should_stop()) {
  6033. +       mutex_lock(&uksm_thread_mutex);
  6034. +       if (ksmd_should_run()) {
  6035. +           uksm_do_scan();
  6036. +       }
  6037. +       mutex_unlock(&uksm_thread_mutex);
  6038. +
  6039. +       try_to_freeze();
  6040. +
  6041. +       if (ksmd_should_run()) {
  6042. +           schedule_timeout_interruptible(uksm_sleep_real);
  6043. +           uksm_sleep_times++;
  6044. +       } else {
  6045. +           wait_event_freezable(uksm_thread_wait,
  6046. +               ksmd_should_run() || kthread_should_stop());
  6047. +       }
  6048. +   }
  6049. +   return 0;
  6050. +}
  6051. +
  6052. +int page_referenced_ksm(struct page *page, struct mem_cgroup *memcg,
  6053. +           unsigned long *vm_flags)
  6054. +{
  6055. +   struct stable_node *stable_node;
  6056. +   struct node_vma *node_vma;
  6057. +   struct rmap_item *rmap_item;
  6058. +   struct hlist_node *hlist, *rmap_hlist;
  6059. +   unsigned int mapcount = page_mapcount(page);
  6060. +   int referenced = 0;
  6061. +   int search_new_forks = 0;
  6062. +   unsigned long address;
  6063. +
  6064. +   VM_BUG_ON(!PageKsm(page));
  6065. +   VM_BUG_ON(!PageLocked(page));
  6066. +
  6067. +   stable_node = page_stable_node(page);
  6068. +   if (!stable_node)
  6069. +       return 0;
  6070. +
  6071. +
  6072. +again:
  6073. +   hlist_for_each_entry(node_vma, hlist, &stable_node->hlist, hlist) {
  6074. +       hlist_for_each_entry(rmap_item, rmap_hlist,
  6075. +                    &node_vma->rmap_hlist, hlist) {
  6076. +           struct anon_vma *anon_vma = rmap_item->anon_vma;
  6077. +           struct anon_vma_chain *vmac;
  6078. +           struct vm_area_struct *vma;
  6079. +
  6080. +           anon_vma_lock(anon_vma);
  6081. +           list_for_each_entry(vmac, &anon_vma->head,
  6082. +                       same_anon_vma) {
  6083. +               vma = vmac->vma;
  6084. +               address = get_rmap_addr(rmap_item);
  6085. +
  6086. +               if (address < vma->vm_start ||
  6087. +                   address >= vma->vm_end)
  6088. +                   continue;
  6089. +               /*
  6090. +                * Initially we examine only the vma which
  6091. +                * covers this rmap_item; but later, if there
  6092. +                * is still work to do, we examine covering
  6093. +                * vmas in other mms: in case they were forked
  6094. +                * from the original since ksmd passed.
  6095. +                */
  6096. +               if ((rmap_item->slot->vma == vma) ==
  6097. +                   search_new_forks)
  6098. +                   continue;
  6099. +
  6100. +               if (memcg &&
  6101. +                   !mm_match_cgroup(vma->vm_mm, memcg))
  6102. +                   continue;
  6103. +
  6104. +               referenced +=
  6105. +                   page_referenced_one(page, vma,
  6106. +                       address, &mapcount, vm_flags);
  6107. +               if (!search_new_forks || !mapcount)
  6108. +                   break;
  6109. +           }
  6110. +
  6111. +           anon_vma_unlock(anon_vma);
  6112. +           if (!mapcount)
  6113. +               goto out;
  6114. +       }
  6115. +   }
  6116. +   if (!search_new_forks++)
  6117. +       goto again;
  6118. +out:
  6119. +   return referenced;
  6120. +}
  6121. +
  6122. +int try_to_unmap_ksm(struct page *page, enum ttu_flags flags)
  6123. +{
  6124. +   struct stable_node *stable_node;
  6125. +   struct node_vma *node_vma;
  6126. +   struct hlist_node *hlist, *rmap_hlist;
  6127. +   struct rmap_item *rmap_item;
  6128. +   int ret = SWAP_AGAIN;
  6129. +   int search_new_forks = 0;
  6130. +   unsigned long address;
  6131. +
  6132. +   VM_BUG_ON(!PageKsm(page));
  6133. +   VM_BUG_ON(!PageLocked(page));
  6134. +
  6135. +   stable_node = page_stable_node(page);
  6136. +   if (!stable_node)
  6137. +       return SWAP_FAIL;
  6138. +again:
  6139. +   hlist_for_each_entry(node_vma, hlist, &stable_node->hlist, hlist) {
  6140. +       hlist_for_each_entry(rmap_item, rmap_hlist,
  6141. +                    &node_vma->rmap_hlist, hlist) {
  6142. +           struct anon_vma *anon_vma = rmap_item->anon_vma;
  6143. +           struct anon_vma_chain *vmac;
  6144. +           struct vm_area_struct *vma;
  6145. +
  6146. +           anon_vma_lock(anon_vma);
  6147. +           list_for_each_entry(vmac, &anon_vma->head,
  6148. +                       same_anon_vma) {
  6149. +               vma = vmac->vma;
  6150. +               address = get_rmap_addr(rmap_item);
  6151. +
  6152. +               if (address < vma->vm_start ||
  6153. +                   address >= vma->vm_end)
  6154. +                   continue;
  6155. +               /*
  6156. +                * Initially we examine only the vma which
  6157. +                * covers this rmap_item; but later, if there
  6158. +                * is still work to do, we examine covering
  6159. +                * vmas in other mms: in case they were forked
  6160. +                * from the original since ksmd passed.
  6161. +                */
  6162. +               if ((rmap_item->slot->vma == vma) ==
  6163. +                   search_new_forks)
  6164. +                   continue;
  6165. +
  6166. +               ret = try_to_unmap_one(page, vma,
  6167. +                              address, flags);
  6168. +               if (ret != SWAP_AGAIN || !page_mapped(page)) {
  6169. +                   anon_vma_unlock(anon_vma);
  6170. +                   goto out;
  6171. +               }
  6172. +           }
  6173. +           anon_vma_unlock(anon_vma);
  6174. +       }
  6175. +   }
  6176. +   if (!search_new_forks++)
  6177. +       goto again;
  6178. +out:
  6179. +   return ret;
  6180. +}
  6181. +
  6182. +#ifdef CONFIG_MIGRATION
  6183. +int rmap_walk_ksm(struct page *page, int (*rmap_one)(struct page *,
  6184. +         struct vm_area_struct *, unsigned long, void *), void *arg)
  6185. +{
  6186. +   struct stable_node *stable_node;
  6187. +   struct node_vma *node_vma;
  6188. +   struct hlist_node *hlist, *rmap_hlist;
  6189. +   struct rmap_item *rmap_item;
  6190. +   int ret = SWAP_AGAIN;
  6191. +   int search_new_forks = 0;
  6192. +   unsigned long address;
  6193. +
  6194. +   VM_BUG_ON(!PageKsm(page));
  6195. +   VM_BUG_ON(!PageLocked(page));
  6196. +
  6197. +   stable_node = page_stable_node(page);
  6198. +   if (!stable_node)
  6199. +       return ret;
  6200. +again:
  6201. +   hlist_for_each_entry(node_vma, hlist, &stable_node->hlist, hlist) {
  6202. +       hlist_for_each_entry(rmap_item, rmap_hlist,
  6203. +                    &node_vma->rmap_hlist, hlist) {
  6204. +           struct anon_vma *anon_vma = rmap_item->anon_vma;
  6205. +           struct anon_vma_chain *vmac;
  6206. +           struct vm_area_struct *vma;
  6207. +
  6208. +           anon_vma_lock(anon_vma);
  6209. +           list_for_each_entry(vmac, &anon_vma->head,
  6210. +                       same_anon_vma) {
  6211. +               vma = vmac->vma;
  6212. +               address = get_rmap_addr(rmap_item);
  6213. +
  6214. +               if (address < vma->vm_start ||
  6215. +                   address >= vma->vm_end)
  6216. +                   continue;
  6217. +
  6218. +               if ((rmap_item->slot->vma == vma) ==
  6219. +                   search_new_forks)
  6220. +                   continue;
  6221. +
  6222. +               ret = rmap_one(page, vma, address, arg);
  6223. +               if (ret != SWAP_AGAIN) {
  6224. +                   anon_vma_unlock(anon_vma);
  6225. +                   goto out;
  6226. +               }
  6227. +           }
  6228. +           anon_vma_unlock(anon_vma);
  6229. +       }
  6230. +   }
  6231. +   if (!search_new_forks++)
  6232. +       goto again;
  6233. +out:
  6234. +   return ret;
  6235. +}
  6236. +
  6237. +/* Common ksm interface but may be specific to uksm */
  6238. +void ksm_migrate_page(struct page *newpage, struct page *oldpage)
  6239. +{
  6240. +   struct stable_node *stable_node;
  6241. +
  6242. +   VM_BUG_ON(!PageLocked(oldpage));
  6243. +   VM_BUG_ON(!PageLocked(newpage));
  6244. +   VM_BUG_ON(newpage->mapping != oldpage->mapping);
  6245. +
  6246. +   stable_node = page_stable_node(newpage);
  6247. +   if (stable_node) {
  6248. +       VM_BUG_ON(stable_node->kpfn != page_to_pfn(oldpage));
  6249. +       stable_node->kpfn = page_to_pfn(newpage);
  6250. +   }
  6251. +}
  6252. +#endif /* CONFIG_MIGRATION */
  6253. +
  6254. +#ifdef CONFIG_MEMORY_HOTREMOVE
  6255. +static struct stable_node *uksm_check_stable_tree(unsigned long start_pfn,
  6256. +                        unsigned long end_pfn)
  6257. +{
  6258. +   struct rb_node *node;
  6259. +
  6260. +   for (node = rb_first(root_stable_treep); node; node = rb_next(node)) {
  6261. +       struct stable_node *stable_node;
  6262. +
  6263. +       stable_node = rb_entry(node, struct stable_node, node);
  6264. +       if (stable_node->kpfn >= start_pfn &&
  6265. +           stable_node->kpfn < end_pfn)
  6266. +           return stable_node;
  6267. +   }
  6268. +   return NULL;
  6269. +}
  6270. +
  6271. +static int uksm_memory_callback(struct notifier_block *self,
  6272. +                  unsigned long action, void *arg)
  6273. +{
  6274. +   struct memory_notify *mn = arg;
  6275. +   struct stable_node *stable_node;
  6276. +
  6277. +   switch (action) {
  6278. +   case MEM_GOING_OFFLINE:
  6279. +       /*
  6280. +        * Keep it very simple for now: just lock out ksmd and
  6281. +        * MADV_UNMERGEABLE while any memory is going offline.
  6282. +        * mutex_lock_nested() is necessary because lockdep was alarmed
  6283. +        * that here we take uksm_thread_mutex inside notifier chain
  6284. +        * mutex, and later take notifier chain mutex inside
  6285. +        * uksm_thread_mutex to unlock it.   But that's safe because both
  6286. +        * are inside mem_hotplug_mutex.
  6287. +        */
  6288. +       mutex_lock_nested(&uksm_thread_mutex, SINGLE_DEPTH_NESTING);
  6289. +       break;
  6290. +
  6291. +   case MEM_OFFLINE:
  6292. +       /*
  6293. +        * Most of the work is done by page migration; but there might
  6294. +        * be a few stable_nodes left over, still pointing to struct
  6295. +        * pages which have been offlined: prune those from the tree.
  6296. +        */
  6297. +       while ((stable_node = uksm_check_stable_tree(mn->start_pfn,
  6298. +                   mn->start_pfn + mn->nr_pages)) != NULL)
  6299. +           remove_node_from_stable_tree(stable_node, 1, 1);
  6300. +       /* fallthrough */
  6301. +
  6302. +   case MEM_CANCEL_OFFLINE:
  6303. +       mutex_unlock(&uksm_thread_mutex);
  6304. +       break;
  6305. +   }
  6306. +   return NOTIFY_OK;
  6307. +}
  6308. +#endif /* CONFIG_MEMORY_HOTREMOVE */
  6309. +
  6310. +#ifdef CONFIG_SYSFS
  6311. +/*
  6312. + * This all compiles without CONFIG_SYSFS, but is a waste of space.
  6313. + */
  6314. +
  6315. +#define UKSM_ATTR_RO(_name) \
  6316. +   static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
  6317. +#define UKSM_ATTR(_name) \
  6318. +   static struct kobj_attribute _name##_attr = \
  6319. +       __ATTR(_name, 0644, _name##_show, _name##_store)
  6320. +
  6321. +static ssize_t max_cpu_percentage_show(struct kobject *kobj,
  6322. +                   struct kobj_attribute *attr, char *buf)
  6323. +{
  6324. +   return sprintf(buf, "%u\n", uksm_max_cpu_percentage);
  6325. +}
  6326. +
  6327. +static ssize_t max_cpu_percentage_store(struct kobject *kobj,
  6328. +                    struct kobj_attribute *attr,
  6329. +                    const char *buf, size_t count)
  6330. +{
  6331. +   unsigned long max_cpu_percentage;
  6332. +   int err;
  6333. +
  6334. +   err = strict_strtoul(buf, 10, &max_cpu_percentage);
  6335. +   if (err || max_cpu_percentage > 100)
  6336. +       return -EINVAL;
  6337. +
  6338. +   if (max_cpu_percentage == 100)
  6339. +       max_cpu_percentage = 99;
  6340. +   else if (max_cpu_percentage < 10)
  6341. +       max_cpu_percentage = 10;
  6342. +
  6343. +   uksm_max_cpu_percentage = max_cpu_percentage;
  6344. +
  6345. +   return count;
  6346. +}
  6347. +UKSM_ATTR(max_cpu_percentage);
  6348. +
  6349. +static ssize_t sleep_millisecs_show(struct kobject *kobj,
  6350. +                   struct kobj_attribute *attr, char *buf)
  6351. +{
  6352. +   return sprintf(buf, "%u\n", jiffies_to_msecs(uksm_sleep_jiffies));
  6353. +}
  6354. +
  6355. +static ssize_t sleep_millisecs_store(struct kobject *kobj,
  6356. +                    struct kobj_attribute *attr,
  6357. +                    const char *buf, size_t count)
  6358. +{
  6359. +   unsigned long msecs;
  6360. +   int err;
  6361. +
  6362. +   err = strict_strtoul(buf, 10, &msecs);
  6363. +   if (err || msecs > MSEC_PER_SEC)
  6364. +       return -EINVAL;
  6365. +
  6366. +   uksm_sleep_jiffies = msecs_to_jiffies(msecs);
  6367. +   uksm_sleep_saved = uksm_sleep_jiffies;
  6368. +
  6369. +   return count;
  6370. +}
  6371. +UKSM_ATTR(sleep_millisecs);
  6372. +
  6373. +
  6374. +static ssize_t cpu_governor_show(struct kobject *kobj,
  6375. +                 struct kobj_attribute *attr, char *buf)
  6376. +{
  6377. +   int n = sizeof(uksm_cpu_governor_str) / sizeof(char *);
  6378. +   int i;
  6379. +
  6380. +   buf[0] = '\0';
  6381. +   for (i = 0; i < n ; i++) {
  6382. +       if (uksm_cpu_governor == i)
  6383. +           strcat(buf, "[");
  6384. +
  6385. +       strcat(buf, uksm_cpu_governor_str[i]);
  6386. +
  6387. +       if (uksm_cpu_governor == i)
  6388. +           strcat(buf, "]");
  6389. +
  6390. +       strcat(buf, " ");
  6391. +   }
  6392. +   strcat(buf, "\n");
  6393. +
  6394. +   return strlen(buf);
  6395. +}
  6396. +
  6397. +static inline void init_performance_values(void)
  6398. +{
  6399. +   int i;
  6400. +   struct scan_rung *rung;
  6401. +   struct uksm_cpu_preset_s *preset = uksm_cpu_preset + uksm_cpu_governor;
  6402. +
  6403. +
  6404. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6405. +       rung = uksm_scan_ladder + i;
  6406. +       rung->cpu_ratio = preset->cpu_ratio[i];
  6407. +       rung->cover_msecs = preset->cover_msecs[i];
  6408. +   }
  6409. +
  6410. +   uksm_max_cpu_percentage = preset->max_cpu;
  6411. +}
  6412. +
  6413. +static ssize_t cpu_governor_store(struct kobject *kobj,
  6414. +                  struct kobj_attribute *attr,
  6415. +                  const char *buf, size_t count)
  6416. +{
  6417. +   int n = sizeof(uksm_cpu_governor_str) / sizeof(char *);
  6418. +
  6419. +   for (n--; n >=0 ; n--) {
  6420. +       if (!strncmp(buf, uksm_cpu_governor_str[n],
  6421. +                strlen(uksm_cpu_governor_str[n])))
  6422. +           break;
  6423. +   }
  6424. +
  6425. +   if (n < 0)
  6426. +       return -EINVAL;
  6427. +   else
  6428. +       uksm_cpu_governor = n;
  6429. +
  6430. +   init_performance_values();
  6431. +
  6432. +   return count;
  6433. +}
  6434. +UKSM_ATTR(cpu_governor);
  6435. +
  6436. +static ssize_t run_show(struct kobject *kobj, struct kobj_attribute *attr,
  6437. +           char *buf)
  6438. +{
  6439. +   return sprintf(buf, "%u\n", uksm_run);
  6440. +}
  6441. +
  6442. +static ssize_t run_store(struct kobject *kobj, struct kobj_attribute *attr,
  6443. +            const char *buf, size_t count)
  6444. +{
  6445. +   int err;
  6446. +   unsigned long flags;
  6447. +
  6448. +   err = strict_strtoul(buf, 10, &flags);
  6449. +   if (err || flags > UINT_MAX)
  6450. +       return -EINVAL;
  6451. +   if (flags > UKSM_RUN_MERGE)
  6452. +       return -EINVAL;
  6453. +
  6454. +   mutex_lock(&uksm_thread_mutex);
  6455. +   if (uksm_run != flags) {
  6456. +       uksm_run = flags;
  6457. +   }
  6458. +   mutex_unlock(&uksm_thread_mutex);
  6459. +
  6460. +   if (flags & UKSM_RUN_MERGE)
  6461. +       wake_up_interruptible(&uksm_thread_wait);
  6462. +
  6463. +   return count;
  6464. +}
  6465. +UKSM_ATTR(run);
  6466. +
  6467. +static ssize_t abundant_threshold_show(struct kobject *kobj,
  6468. +                    struct kobj_attribute *attr, char *buf)
  6469. +{
  6470. +   return sprintf(buf, "%u\n", uksm_abundant_threshold);
  6471. +}
  6472. +
  6473. +static ssize_t abundant_threshold_store(struct kobject *kobj,
  6474. +                     struct kobj_attribute *attr,
  6475. +                     const char *buf, size_t count)
  6476. +{
  6477. +   int err;
  6478. +   unsigned long flags;
  6479. +
  6480. +   err = strict_strtoul(buf, 10, &flags);
  6481. +   if (err || flags > 99)
  6482. +       return -EINVAL;
  6483. +
  6484. +   uksm_abundant_threshold = flags;
  6485. +
  6486. +   return count;
  6487. +}
  6488. +UKSM_ATTR(abundant_threshold);
  6489. +
  6490. +static ssize_t thrash_threshold_show(struct kobject *kobj,
  6491. +                    struct kobj_attribute *attr, char *buf)
  6492. +{
  6493. +   return sprintf(buf, "%u\n", uksm_thrash_threshold);
  6494. +}
  6495. +
  6496. +static ssize_t thrash_threshold_store(struct kobject *kobj,
  6497. +                     struct kobj_attribute *attr,
  6498. +                     const char *buf, size_t count)
  6499. +{
  6500. +   int err;
  6501. +   unsigned long flags;
  6502. +
  6503. +   err = strict_strtoul(buf, 10, &flags);
  6504. +   if (err || flags > 99)
  6505. +       return -EINVAL;
  6506. +
  6507. +   uksm_thrash_threshold = flags;
  6508. +
  6509. +   return count;
  6510. +}
  6511. +UKSM_ATTR(thrash_threshold);
  6512. +
  6513. +static ssize_t cpu_ratios_show(struct kobject *kobj,
  6514. +                  struct kobj_attribute *attr, char *buf)
  6515. +{
  6516. +   int i, size;
  6517. +   struct scan_rung *rung;
  6518. +   char *p = buf;
  6519. +
  6520. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6521. +       rung = &uksm_scan_ladder[i];
  6522. +
  6523. +       if (rung->cpu_ratio > 0)
  6524. +           size = sprintf(p, "%d ", rung->cpu_ratio);
  6525. +       else
  6526. +           size = sprintf(p, "MAX/%d ",
  6527. +                   TIME_RATIO_SCALE / -rung->cpu_ratio);
  6528. +
  6529. +       p += size;
  6530. +   }
  6531. +
  6532. +   *p++ = '\n';
  6533. +   *p = '\0';
  6534. +
  6535. +   return p - buf;
  6536. +}
  6537. +
  6538. +static ssize_t cpu_ratios_store(struct kobject *kobj,
  6539. +                     struct kobj_attribute *attr,
  6540. +                     const char *buf, size_t count)
  6541. +{
  6542. +   int i, cpuratios[SCAN_LADDER_SIZE], err;
  6543. +   unsigned long value;
  6544. +   struct scan_rung *rung;
  6545. +   char *p, *end = NULL;
  6546. +
  6547. +   p = kzalloc(count, GFP_KERNEL);
  6548. +   if (!p)
  6549. +       return -ENOMEM;
  6550. +
  6551. +   memcpy(p, buf, count);
  6552. +
  6553. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6554. +       if (i != SCAN_LADDER_SIZE -1) {
  6555. +           end = strchr(p, ' ');
  6556. +           if (!end)
  6557. +               return -EINVAL;
  6558. +
  6559. +           *end = '\0';
  6560. +       }
  6561. +
  6562. +       if (strstr(p, "MAX/")) {
  6563. +           p = strchr(p, '/') + 1;
  6564. +           err = strict_strtoul(p, 10, &value);
  6565. +           if (err || value > TIME_RATIO_SCALE || !value)
  6566. +               return -EINVAL;
  6567. +
  6568. +           cpuratios[i] = - (int) (TIME_RATIO_SCALE / value);
  6569. +       } else {
  6570. +           err = strict_strtoul(p, 10, &value);
  6571. +           if (err || value > TIME_RATIO_SCALE || !value)
  6572. +               return -EINVAL;
  6573. +
  6574. +           cpuratios[i] = value;
  6575. +       }
  6576. +
  6577. +       p = end + 1;
  6578. +   }
  6579. +
  6580. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6581. +       rung = &uksm_scan_ladder[i];
  6582. +
  6583. +       rung->cpu_ratio = cpuratios[i];
  6584. +   }
  6585. +
  6586. +   return count;
  6587. +}
  6588. +UKSM_ATTR(cpu_ratios);
  6589. +
  6590. +static ssize_t eval_intervals_show(struct kobject *kobj,
  6591. +                  struct kobj_attribute *attr, char *buf)
  6592. +{
  6593. +   int i, size;
  6594. +   struct scan_rung *rung;
  6595. +   char *p = buf;
  6596. +
  6597. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6598. +       rung = &uksm_scan_ladder[i];
  6599. +       size = sprintf(p, "%u ", rung->cover_msecs);
  6600. +       p += size;
  6601. +   }
  6602. +
  6603. +   *p++ = '\n';
  6604. +   *p = '\0';
  6605. +
  6606. +   return p - buf;
  6607. +}
  6608. +
  6609. +static ssize_t eval_intervals_store(struct kobject *kobj,
  6610. +                     struct kobj_attribute *attr,
  6611. +                     const char *buf, size_t count)
  6612. +{
  6613. +   int i, err;
  6614. +   unsigned long values[SCAN_LADDER_SIZE];
  6615. +   struct scan_rung *rung;
  6616. +   char *p, *end = NULL;
  6617. +
  6618. +   p = kzalloc(count, GFP_KERNEL);
  6619. +   if (!p)
  6620. +       return -ENOMEM;
  6621. +
  6622. +   memcpy(p, buf, count);
  6623. +
  6624. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6625. +       if (i != SCAN_LADDER_SIZE -1) {
  6626. +           end = strchr(p, ' ');
  6627. +           if (!end)
  6628. +               return -EINVAL;
  6629. +
  6630. +           *end = '\0';
  6631. +       }
  6632. +
  6633. +       err = strict_strtoul(p, 10, &values[i]);
  6634. +       if (err)
  6635. +           return -EINVAL;
  6636. +
  6637. +       p = end + 1;
  6638. +   }
  6639. +
  6640. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6641. +       rung = &uksm_scan_ladder[i];
  6642. +
  6643. +       rung->cover_msecs = values[i];
  6644. +   }
  6645. +
  6646. +   return count;
  6647. +}
  6648. +UKSM_ATTR(eval_intervals);
  6649. +
  6650. +static ssize_t ema_per_page_time_show(struct kobject *kobj,
  6651. +                struct kobj_attribute *attr, char *buf)
  6652. +{
  6653. +   return sprintf(buf, "%lu\n", uksm_ema_page_time);
  6654. +}
  6655. +UKSM_ATTR_RO(ema_per_page_time);
  6656. +
  6657. +static ssize_t pages_shared_show(struct kobject *kobj,
  6658. +                struct kobj_attribute *attr, char *buf)
  6659. +{
  6660. +   return sprintf(buf, "%lu\n", uksm_pages_shared);
  6661. +}
  6662. +UKSM_ATTR_RO(pages_shared);
  6663. +
  6664. +static ssize_t pages_sharing_show(struct kobject *kobj,
  6665. +                 struct kobj_attribute *attr, char *buf)
  6666. +{
  6667. +   return sprintf(buf, "%lu\n", uksm_pages_sharing);
  6668. +}
  6669. +UKSM_ATTR_RO(pages_sharing);
  6670. +
  6671. +static ssize_t pages_unshared_show(struct kobject *kobj,
  6672. +                  struct kobj_attribute *attr, char *buf)
  6673. +{
  6674. +   return sprintf(buf, "%lu\n", uksm_pages_unshared);
  6675. +}
  6676. +UKSM_ATTR_RO(pages_unshared);
  6677. +
  6678. +static ssize_t full_scans_show(struct kobject *kobj,
  6679. +                  struct kobj_attribute *attr, char *buf)
  6680. +{
  6681. +   return sprintf(buf, "%llu\n", fully_scanned_round);
  6682. +}
  6683. +UKSM_ATTR_RO(full_scans);
  6684. +
  6685. +static ssize_t pages_scanned_show(struct kobject *kobj,
  6686. +                 struct kobj_attribute *attr, char *buf)
  6687. +{
  6688. +   unsigned long base = 0;
  6689. +   u64 delta, ret;
  6690. +
  6691. +   if (pages_scanned_stored) {
  6692. +       base = pages_scanned_base;
  6693. +       ret = pages_scanned_stored;
  6694. +       delta = uksm_pages_scanned >> base;
  6695. +       if (CAN_OVERFLOW_U64(ret, delta)) {
  6696. +           ret >>= 1;
  6697. +           delta >>= 1;
  6698. +           base++;
  6699. +           ret += delta;
  6700. +       }
  6701. +   } else {
  6702. +       ret = uksm_pages_scanned;
  6703. +   }
  6704. +
  6705. +   while (ret > ULONG_MAX) {
  6706. +       ret >>= 1;
  6707. +       base++;
  6708. +   }
  6709. +
  6710. +   if (base)
  6711. +       return sprintf(buf, "%lu * 2^%lu\n", (unsigned long)ret, base);
  6712. +   else
  6713. +       return sprintf(buf, "%lu\n", (unsigned long)ret);
  6714. +}
  6715. +UKSM_ATTR_RO(pages_scanned);
  6716. +
  6717. +static ssize_t hash_strength_show(struct kobject *kobj,
  6718. +                 struct kobj_attribute *attr, char *buf)
  6719. +{
  6720. +   return sprintf(buf, "%lu\n", hash_strength);
  6721. +}
  6722. +UKSM_ATTR_RO(hash_strength);
  6723. +
  6724. +static ssize_t sleep_times_show(struct kobject *kobj,
  6725. +                 struct kobj_attribute *attr, char *buf)
  6726. +{
  6727. +   return sprintf(buf, "%llu\n", uksm_sleep_times);
  6728. +}
  6729. +UKSM_ATTR_RO(sleep_times);
  6730. +
  6731. +
  6732. +static struct attribute *uksm_attrs[] = {
  6733. +   &max_cpu_percentage_attr.attr,
  6734. +   &sleep_millisecs_attr.attr,
  6735. +   &cpu_governor_attr.attr,
  6736. +   &run_attr.attr,
  6737. +   &ema_per_page_time_attr.attr,
  6738. +   &pages_shared_attr.attr,
  6739. +   &pages_sharing_attr.attr,
  6740. +   &pages_unshared_attr.attr,
  6741. +   &full_scans_attr.attr,
  6742. +   &pages_scanned_attr.attr,
  6743. +   &hash_strength_attr.attr,
  6744. +   &sleep_times_attr.attr,
  6745. +   &thrash_threshold_attr.attr,
  6746. +   &abundant_threshold_attr.attr,
  6747. +   &cpu_ratios_attr.attr,
  6748. +   &eval_intervals_attr.attr,
  6749. +   NULL,
  6750. +};
  6751. +
  6752. +static struct attribute_group uksm_attr_group = {
  6753. +   .attrs = uksm_attrs,
  6754. +   .name = "uksm",
  6755. +};
  6756. +#endif /* CONFIG_SYSFS */
  6757. +
  6758. +static inline void init_scan_ladder(void)
  6759. +{
  6760. +   int i;
  6761. +   struct scan_rung *rung;
  6762. +
  6763. +   for (i = 0; i < SCAN_LADDER_SIZE; i++) {
  6764. +       rung = uksm_scan_ladder + i;
  6765. +       slot_tree_init_root(&rung->vma_root);
  6766. +   }
  6767. +
  6768. +   init_performance_values();
  6769. +   uksm_calc_scan_pages();
  6770. +}
  6771. +
  6772. +static inline int cal_positive_negative_costs(void)
  6773. +{
  6774. +   struct page *p1, *p2;
  6775. +   unsigned char *addr1, *addr2;
  6776. +   unsigned long i, time_start, hash_cost;
  6777. +   unsigned long loopnum = 0;
  6778. +
  6779. +   /*IMPORTANT: volatile is needed to prevent over-optimization by gcc. */
  6780. +   volatile u32 hash;
  6781. +   volatile int ret;
  6782. +
  6783. +   p1 = alloc_page(GFP_KERNEL);
  6784. +   if (!p1)
  6785. +       return -ENOMEM;
  6786. +
  6787. +   p2 = alloc_page(GFP_KERNEL);
  6788. +   if (!p2)
  6789. +       return -ENOMEM;
  6790. +
  6791. +   addr1 = kmap_atomic(p1);
  6792. +   addr2 = kmap_atomic(p2);
  6793. +   memset(addr1, random32(), PAGE_SIZE);
  6794. +   memcpy(addr2, addr1, PAGE_SIZE);
  6795. +
  6796. +   /* make sure that the two pages differ in last byte */
  6797. +   addr2[PAGE_SIZE-1] = ~addr2[PAGE_SIZE-1];
  6798. +   kunmap_atomic(addr2);
  6799. +   kunmap_atomic(addr1);
  6800. +
  6801. +   time_start = jiffies;
  6802. +   while (jiffies - time_start < 100) {
  6803. +       for (i = 0; i < 100; i++)
  6804. +           hash = page_hash(p1, HASH_STRENGTH_FULL, 0);
  6805. +       loopnum += 100;
  6806. +   }
  6807. +   hash_cost = (jiffies - time_start);
  6808. +
  6809. +   time_start = jiffies;
  6810. +   for (i = 0; i < loopnum; i++)
  6811. +       ret = pages_identical(p1, p2);
  6812. +   memcmp_cost = HASH_STRENGTH_FULL * (jiffies - time_start);
  6813. +   memcmp_cost /= hash_cost;
  6814. +   printk(KERN_INFO "UKSM: relative memcmp_cost = %lu "
  6815. +            "hash=%u cmp_ret=%d.\n",
  6816. +          memcmp_cost, hash, ret);
  6817. +
  6818. +   __free_page(p1);
  6819. +   __free_page(p2);
  6820. +   return 0;
  6821. +}
  6822. +
  6823. +static int init_zeropage_hash_table(void)
  6824. +{
  6825. +   struct page *page;
  6826. +   char *addr;
  6827. +   int i;
  6828. +
  6829. +   page = alloc_page(GFP_KERNEL);
  6830. +   if (!page)
  6831. +       return -ENOMEM;
  6832. +
  6833. +   addr = kmap_atomic(page);
  6834. +   memset(addr, 0, PAGE_SIZE);
  6835. +   kunmap_atomic(addr);
  6836. +
  6837. +   zero_hash_table = kmalloc(HASH_STRENGTH_MAX * sizeof(u32),
  6838. +       GFP_KERNEL);
  6839. +   if (!zero_hash_table)
  6840. +       return -ENOMEM;
  6841. +
  6842. +   for (i = 0; i < HASH_STRENGTH_MAX; i++)
  6843. +       zero_hash_table[i] = page_hash(page, i, 0);
  6844. +
  6845. +   __free_page(page);
  6846. +
  6847. +   return 0;
  6848. +}
  6849. +
  6850. +static inline int init_random_sampling(void)
  6851. +{
  6852. +   unsigned long i;
  6853. +   random_nums = kmalloc(PAGE_SIZE, GFP_KERNEL);
  6854. +   if (!random_nums)
  6855. +       return -ENOMEM;
  6856. +
  6857. +   for (i = 0; i < HASH_STRENGTH_FULL; i++)
  6858. +       random_nums[i] = i;
  6859. +
  6860. +   for (i = 0; i < HASH_STRENGTH_FULL; i++) {
  6861. +       unsigned long rand_range, swap_index, tmp;
  6862. +
  6863. +       rand_range = HASH_STRENGTH_FULL - i;
  6864. +       swap_index = i + random32() % rand_range;
  6865. +       tmp = random_nums[i];
  6866. +       random_nums[i] =  random_nums[swap_index];
  6867. +       random_nums[swap_index] = tmp;
  6868. +   }
  6869. +
  6870. +   rshash_state.state = RSHASH_NEW;
  6871. +   rshash_state.below_count = 0;
  6872. +   rshash_state.lookup_window_index = 0;
  6873. +
  6874. +   return cal_positive_negative_costs();
  6875. +}
  6876. +
  6877. +static int __init uksm_slab_init(void)
  6878. +{
  6879. +   rmap_item_cache = UKSM_KMEM_CACHE(rmap_item, 0);
  6880. +   if (!rmap_item_cache)
  6881. +       goto out;
  6882. +
  6883. +   stable_node_cache = UKSM_KMEM_CACHE(stable_node, 0);
  6884. +   if (!stable_node_cache)
  6885. +       goto out_free1;
  6886. +
  6887. +   node_vma_cache = UKSM_KMEM_CACHE(node_vma, 0);
  6888. +   if (!node_vma_cache)
  6889. +       goto out_free2;
  6890. +
  6891. +   vma_slot_cache = UKSM_KMEM_CACHE(vma_slot, 0);
  6892. +   if (!vma_slot_cache)
  6893. +       goto out_free3;
  6894. +
  6895. +   tree_node_cache = UKSM_KMEM_CACHE(tree_node, 0);
  6896. +   if (!tree_node_cache)
  6897. +       goto out_free4;
  6898. +
  6899. +   return 0;
  6900. +
  6901. +out_free4:
  6902. +   kmem_cache_destroy(vma_slot_cache);
  6903. +out_free3:
  6904. +   kmem_cache_destroy(node_vma_cache);
  6905. +out_free2:
  6906. +   kmem_cache_destroy(stable_node_cache);
  6907. +out_free1:
  6908. +   kmem_cache_destroy(rmap_item_cache);
  6909. +out:
  6910. +   return -ENOMEM;
  6911. +}
  6912. +
  6913. +static void __init uksm_slab_free(void)
  6914. +{
  6915. +   kmem_cache_destroy(stable_node_cache);
  6916. +   kmem_cache_destroy(rmap_item_cache);
  6917. +   kmem_cache_destroy(node_vma_cache);
  6918. +   kmem_cache_destroy(vma_slot_cache);
  6919. +   kmem_cache_destroy(tree_node_cache);
  6920. +}
  6921. +
  6922. +/* Common interface to ksm, different to it. */
  6923. +int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
  6924. +       unsigned long end, int advice, unsigned long *vm_flags)
  6925. +{
  6926. +   int err;
  6927. +
  6928. +   switch (advice) {
  6929. +   case MADV_MERGEABLE:
  6930. +       return 0;       /* just ignore the advice */
  6931. +
  6932. +   case MADV_UNMERGEABLE:
  6933. +       if (!(*vm_flags & VM_MERGEABLE))
  6934. +           return 0;       /* just ignore the advice */
  6935. +
  6936. +       if (vma->anon_vma) {
  6937. +           err = unmerge_uksm_pages(vma, start, end);
  6938. +           if (err)
  6939. +               return err;
  6940. +       }
  6941. +
  6942. +       uksm_remove_vma(vma);
  6943. +       *vm_flags &= ~VM_MERGEABLE;
  6944. +       break;
  6945. +   }
  6946. +
  6947. +   return 0;
  6948. +}
  6949. +
  6950. +/* Common interface to ksm, actually the same. */
  6951. +struct page *ksm_does_need_to_copy(struct page *page,
  6952. +           struct vm_area_struct *vma, unsigned long address)
  6953. +{
  6954. +   struct page *new_page;
  6955. +
  6956. +   new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, address);
  6957. +   if (new_page) {
  6958. +       copy_user_highpage(new_page, page, address, vma);
  6959. +
  6960. +       SetPageDirty(new_page);
  6961. +       __SetPageUptodate(new_page);
  6962. +       SetPageSwapBacked(new_page);
  6963. +       __set_page_locked(new_page);
  6964. +
  6965. +       if (page_evictable(new_page, vma))
  6966. +           lru_cache_add_lru(new_page, LRU_ACTIVE_ANON);
  6967. +       else
  6968. +           add_page_to_unevictable_list(new_page);
  6969. +   }
  6970. +
  6971. +   return new_page;
  6972. +}
  6973. +
  6974. +static int __init uksm_init(void)
  6975. +{
  6976. +   struct task_struct *uksm_thread;
  6977. +   int err;
  6978. +
  6979. +   uksm_sleep_jiffies = msecs_to_jiffies(100);
  6980. +   uksm_sleep_saved = uksm_sleep_jiffies;
  6981. +
  6982. +   slot_tree_init();
  6983. +   init_scan_ladder();
  6984. +
  6985. +
  6986. +   err = init_random_sampling();
  6987. +   if (err)
  6988. +       goto out_free2;
  6989. +
  6990. +   err = uksm_slab_init();
  6991. +   if (err)
  6992. +       goto out_free1;
  6993. +
  6994. +   err = init_zeropage_hash_table();
  6995. +   if (err)
  6996. +       goto out_free0;
  6997. +
  6998. +   uksm_thread = kthread_run(uksm_scan_thread, NULL, "uksmd");
  6999. +   if (IS_ERR(uksm_thread)) {
  7000. +       printk(KERN_ERR "uksm: creating kthread failed\n");
  7001. +       err = PTR_ERR(uksm_thread);
  7002. +       goto out_free;
  7003. +   }
  7004. +
  7005. +#ifdef CONFIG_SYSFS
  7006. +   err = sysfs_create_group(mm_kobj, &uksm_attr_group);
  7007. +   if (err) {
  7008. +       printk(KERN_ERR "uksm: register sysfs failed\n");
  7009. +       kthread_stop(uksm_thread);
  7010. +       goto out_free;
  7011. +   }
  7012. +#else
  7013. +   uksm_run = UKSM_RUN_MERGE;  /* no way for user to start it */
  7014. +
  7015. +#endif /* CONFIG_SYSFS */
  7016. +
  7017. +#ifdef CONFIG_MEMORY_HOTREMOVE
  7018. +   /*
  7019. +    * Choose a high priority since the callback takes uksm_thread_mutex:
  7020. +    * later callbacks could only be taking locks which nest within that.
  7021. +    */
  7022. +   hotplug_memory_notifier(uksm_memory_callback, 100);
  7023. +#endif
  7024. +   return 0;
  7025. +
  7026. +out_free:
  7027. +   kfree(zero_hash_table);
  7028. +out_free0:
  7029. +   uksm_slab_free();
  7030. +out_free1:
  7031. +   kfree(random_nums);
  7032. +out_free2:
  7033. +   kfree(uksm_scan_ladder);
  7034. +   return err;
  7035. +}
  7036. +
  7037. +#ifdef MODULE
  7038. +module_init(uksm_init)
  7039. +#else
  7040. +late_initcall(uksm_init);
  7041. +#endif
  7042. +
  7043. diff --git a/mm/vmstat.c b/mm/vmstat.c
  7044. index cec1553..32f8279 100644
  7045. --- a/mm/vmstat.c
  7046. +++ b/mm/vmstat.c
  7047. @@ -726,6 +726,9 @@ const char * const vmstat_text[] = {
  7048.  #endif
  7049.     "nr_anon_transparent_hugepages",
  7050.     "nr_free_cma",
  7051. +#ifdef CONFIG_UKSM
  7052. +   "nr_uksm_zero_pages",
  7053. +#endif
  7054.     "nr_dirty_threshold",
  7055.     "nr_dirty_background_threshold",
  7056.  
  7057. --
  7058. 2.4.0
  7059.  
  7060. From a593a8ce7b92a1ed6d5d0ca7b7172e62b00bb58a Mon Sep 17 00:00:00 2001
  7061. From: luca020400 <[email protected]>
  7062. Date: Wed, 6 May 2015 11:56:41 +0200
  7063. Subject: [PATCH 2/2] falcon peregrine titan: enable UKSM
  7064.  
  7065. Change-Id: Ia4b1b308d6edeb4612126370f52915586d3edaf6
  7066. ---
  7067. arch/arm/configs/falcon_defconfig    | 3 ++-
  7068.  arch/arm/configs/peregrine_defconfig | 3 ++-
  7069.  arch/arm/configs/titan_defconfig     | 3 ++-
  7070.  3 files changed, 6 insertions(+), 3 deletions(-)
  7071.  
  7072. diff --git a/arch/arm/configs/falcon_defconfig b/arch/arm/configs/falcon_defconfig
  7073. index e730f06..aa8a4eb 100644
  7074. --- a/arch/arm/configs/falcon_defconfig
  7075. +++ b/arch/arm/configs/falcon_defconfig
  7076. @@ -643,7 +643,8 @@ CONFIG_MIGRATION=y
  7077.  CONFIG_ZONE_DMA_FLAG=0
  7078.  CONFIG_BOUNCE=y
  7079.  CONFIG_VIRT_TO_BUS=y
  7080. -CONFIG_KSM=y
  7081. +# CONFIG_KSM_LEGACY is not set
  7082. +CONFIG_UKSM=y
  7083.  CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
  7084.  # CONFIG_CLEANCACHE is not set
  7085.  CONFIG_MEMORY_HOLE_CARVEOUT=y
  7086. diff --git a/arch/arm/configs/peregrine_defconfig b/arch/arm/configs/peregrine_defconfig
  7087. index 62394a9..bc4a280 100644
  7088. --- a/arch/arm/configs/peregrine_defconfig
  7089. +++ b/arch/arm/configs/peregrine_defconfig
  7090. @@ -643,7 +643,8 @@ CONFIG_MIGRATION=y
  7091.  CONFIG_ZONE_DMA_FLAG=0
  7092.  CONFIG_BOUNCE=y
  7093.  CONFIG_VIRT_TO_BUS=y
  7094. -CONFIG_KSM=y
  7095. +# CONFIG_KSM_LEGACY is not set
  7096. +CONFIG_UKSM=y
  7097.  CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
  7098.  # CONFIG_CLEANCACHE is not set
  7099.  CONFIG_MEMORY_HOLE_CARVEOUT=y
  7100. diff --git a/arch/arm/configs/titan_defconfig b/arch/arm/configs/titan_defconfig
  7101. index 7dd46db..36de795 100644
  7102. --- a/arch/arm/configs/titan_defconfig
  7103. +++ b/arch/arm/configs/titan_defconfig
  7104. @@ -643,7 +643,8 @@ CONFIG_MIGRATION=y
  7105.  CONFIG_ZONE_DMA_FLAG=0
  7106.  CONFIG_BOUNCE=y
  7107.  CONFIG_VIRT_TO_BUS=y
  7108. -CONFIG_KSM=y
  7109. +# CONFIG_KSM_LEGACY is not set
  7110. +CONFIG_UKSM=y
  7111.  CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
  7112.  # CONFIG_CLEANCACHE is not set
  7113.  CONFIG_MEMORY_HOLE_CARVEOUT=y
  7114. --
  7115. 2.4.0
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!